Skip to content

Commit a6e2aa7

Browse files
committed
Add more description of parameter for Netty
1 parent 13e610b commit a6e2aa7

File tree

1 file changed

+41
-0
lines changed

1 file changed

+41
-0
lines changed

docs/configuration.md

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -864,6 +864,47 @@ Apart from these, the following properties are also available, and may be useful
864864
between nodes leading to flooding the network with those.
865865
</td>
866866
</tr>
867+
<tr>
868+
<td><code>spark.shuffle.io.mode</code></td>
869+
<td>NIO</td>
870+
<td>
871+
(Netty only) IO mode for channel implementation. If you set <code>NIO</code>, pure java socket is used for the channel.
872+
Else if you set <code>EPOLL</code>, higher performance native socket is used through JNI.
873+
</td>
874+
</tr>
875+
<tr>
876+
<td><code>spark.shuffle.io.backLog</code></td>
877+
<td>-1</td>
878+
<td>
879+
(Netty only) Maximum length of the queue incomming connections.
880+
If this value &lt;= 0, the default length depending on your platform is used.
881+
<code>200</code> is set for Windows, and <code>128</code> is set for others.
882+
Especially, your platform has <code>/proc/sys/net/core/somaxconn</code>,
883+
the value recorded in this file is preferred.
884+
</td>
885+
</tr>
886+
<tr>
887+
<td><code>spark.shuffle.io.receiveBuffer</code></td>
888+
<td>-1</td>
889+
<td>
890+
(Netty only) Receive buffer size for transportation.
891+
If this value &lt;= 0, the default value depending on your platform and
892+
IO mode represented by <code>spark.shuffle.io.mode</code> is used.
893+
The optimal size for receive buffer should be
894+
<code>latency * network_bandwidth</code>. Assuming latency = 1ms, network_bandwidth = 10Gbps,
895+
buffer size should be ~ 1.25MB.
896+
</td>
897+
</tr>
898+
<tr>
899+
<td><code>spark.shuffle.io.sendBuffer</code></td>
900+
<td>-1</td>
901+
<td>
902+
(Netty only) Send buffer size for transportation.
903+
If this value &lt;= 0, the default value is set as with <code>spark.shuffle.io.receiveBuffer</code>.
904+
As well as receive buffer, the optimal size for send buffer should be
905+
considered latency and network bandwidth.
906+
</td>
907+
</tr>
867908
<tr>
868909
<td><code>spark.shuffle.io.preferDirectBufs</code></td>
869910
<td>true</td>

0 commit comments

Comments
 (0)