-
- Downloads
[SPARK-4740] Create multiple concurrent connections between two peer nodes in Netty.
It's been reported that when the number of disks is large and the number of nodes is small, Netty network throughput is low compared with NIO. We suspect the problem is that only a small number of disks are utilized to serve shuffle files at any given point, due to connection reuse. This patch adds a new config parameter to specify the number of concurrent connections between two peer nodes, default to 2. Author: Reynold Xin <rxin@databricks.com> Closes #3625 from rxin/SPARK-4740 and squashes the following commits: ad4241a [Reynold Xin] Updated javadoc. f33c72b [Reynold Xin] Code review feedback. 0fefabb [Reynold Xin] Use double check in synchronization. 41dfcb2 [Reynold Xin] Added test case. 9076b4a [Reynold Xin] Fixed two NPEs. 3e1306c [Reynold Xin] Minor style fix. 4f21673 [Reynold Xin] [SPARK-4740] Create multiple concurrent connections between two peer nodes in Netty.
Showing
- network/common/src/main/java/org/apache/spark/network/client/TransportClientFactory.java 85 additions, 39 deletions...g/apache/spark/network/client/TransportClientFactory.java
- network/common/src/main/java/org/apache/spark/network/util/TransportConf.java 5 additions, 0 deletions...ain/java/org/apache/spark/network/util/TransportConf.java
- network/common/src/test/java/org/apache/spark/network/TransportClientFactorySuite.java 90 additions, 7 deletions...org/apache/spark/network/TransportClientFactorySuite.java
Loading
Please register or sign in to comment