Skip to content
Snippets Groups Projects
Commit fbca6b6c authored by Aaron Davidson's avatar Aaron Davidson Committed by Patrick Wendell
Browse files

[SPARK-4864] Add documentation to Netty-based configs

Author: Aaron Davidson <aaron@databricks.com>

Closes #3713 from aarondav/netty-configs and squashes the following commits:

8a8b373 [Aaron Davidson] Address Patrick's comments
3b1f84e [Aaron Davidson] [SPARK-4864] Add documentation to Netty-based configs
parent 7c0ed13d
No related branches found
No related tags found
No related merge requests found
......@@ -852,6 +852,41 @@ Apart from these, the following properties are also available, and may be useful
between nodes leading to flooding the network with those.
</td>
</tr>
<tr>
<td><code>spark.shuffle.io.preferDirectBufs</code></td>
<td>true</td>
<td>
(Netty only) Off-heap buffers are used to reduce garbage collection during shuffle and cache
block transfer. For environments where off-heap memory is tightly limited, users may wish to
turn this off to force all allocations from Netty to be on-heap.
</td>
</tr>
<tr>
<td><code>spark.shuffle.io.numConnectionsPerPeer</code></td>
<td>1</td>
<td>
(Netty only) Connections between hosts are reused in order to reduce connection buildup for
large clusters. For clusters with many hard disks and few hosts, this may result in insufficient
concurrency to saturate all disks, and so users may consider increasing this value.
</td>
</tr>
<tr>
<td><code>spark.shuffle.io.maxRetries</code></td>
<td>3</td>
<td>
(Netty only) Fetches that fail due to IO-related exceptions are automatically retried if this is
set to a non-zero value. This retry logic helps stabilize large shuffles in the face of long GC
pauses or transient network connectivity issues.
</td>
</tr>
<tr>
<td><code>spark.shuffle.io.retryWait</code></td>
<td>5</td>
<td>
(Netty only) Seconds to wait between retries of fetches. The maximum delay caused by retrying
is simply <code>maxRetries * retryWait</code>, by default 15 seconds.
</td>
</tr>
</table>
#### Scheduling
......
......@@ -40,7 +40,7 @@ public class TransportConf {
return conf.getInt("spark.shuffle.io.connectionTimeout", 120) * 1000;
}
/** Number of concurrent connections between two nodes for fetching data. **/
/** Number of concurrent connections between two nodes for fetching data. */
public int numConnectionsPerPeer() {
return conf.getInt("spark.shuffle.io.numConnectionsPerPeer", 1);
}
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment