-
- Downloads
[SPARK-10745][CORE] Separate configs between shuffle and RPC
[SPARK-6028](https://issues.apache.org/jira/browse/SPARK-6028) uses network module to implement RPC. However, there are some configurations named with `spark.shuffle` prefix in the network module. This PR refactors them to make sure the user can control them in shuffle and RPC separately. The user can use `spark.rpc.*` to set the configuration for netty RPC. Author: Shixiong Zhu <shixiong@databricks.com> Closes #9481 from zsxwing/SPARK-10745.
Showing
- core/src/main/scala/org/apache/spark/deploy/ExternalShuffleService.scala 2 additions, 1 deletion...cala/org/apache/spark/deploy/ExternalShuffleService.scala
- core/src/main/scala/org/apache/spark/network/netty/NettyBlockTransferService.scala 1 addition, 1 deletion...pache/spark/network/netty/NettyBlockTransferService.scala
- core/src/main/scala/org/apache/spark/network/netty/SparkTransportConf.scala 6 additions, 6 deletions...a/org/apache/spark/network/netty/SparkTransportConf.scala
- core/src/main/scala/org/apache/spark/rpc/netty/NettyRpcEnv.scala 3 additions, 5 deletions...c/main/scala/org/apache/spark/rpc/netty/NettyRpcEnv.scala
- core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala 1 addition, 1 deletion...scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala
- core/src/main/scala/org/apache/spark/shuffle/FileShuffleBlockResolver.scala 1 addition, 1 deletion...a/org/apache/spark/shuffle/FileShuffleBlockResolver.scala
- core/src/main/scala/org/apache/spark/shuffle/IndexShuffleBlockResolver.scala 1 addition, 1 deletion.../org/apache/spark/shuffle/IndexShuffleBlockResolver.scala
- core/src/main/scala/org/apache/spark/storage/BlockManager.scala 1 addition, 1 deletion...rc/main/scala/org/apache/spark/storage/BlockManager.scala
- core/src/test/scala/org/apache/spark/ExternalShuffleServiceSuite.scala 1 addition, 1 deletion.../scala/org/apache/spark/ExternalShuffleServiceSuite.scala
- network/common/src/main/java/org/apache/spark/network/util/TransportConf.java 50 additions, 15 deletions...ain/java/org/apache/spark/network/util/TransportConf.java
- network/common/src/test/java/org/apache/spark/network/ChunkFetchIntegrationSuite.java 1 addition, 1 deletion.../org/apache/spark/network/ChunkFetchIntegrationSuite.java
- network/common/src/test/java/org/apache/spark/network/RequestTimeoutIntegrationSuite.java 1 addition, 1 deletion.../apache/spark/network/RequestTimeoutIntegrationSuite.java
- network/common/src/test/java/org/apache/spark/network/RpcIntegrationSuite.java 1 addition, 1 deletion...st/java/org/apache/spark/network/RpcIntegrationSuite.java
- network/common/src/test/java/org/apache/spark/network/StreamSuite.java 1 addition, 1 deletion...n/src/test/java/org/apache/spark/network/StreamSuite.java
- network/common/src/test/java/org/apache/spark/network/TransportClientFactorySuite.java 3 additions, 3 deletions...org/apache/spark/network/TransportClientFactorySuite.java
- network/common/src/test/java/org/apache/spark/network/sasl/SparkSaslSuite.java 3 additions, 3 deletions...st/java/org/apache/spark/network/sasl/SparkSaslSuite.java
- network/shuffle/src/test/java/org/apache/spark/network/sasl/SaslIntegrationSuite.java 1 addition, 1 deletion...a/org/apache/spark/network/sasl/SaslIntegrationSuite.java
- network/shuffle/src/test/java/org/apache/spark/network/shuffle/ExternalShuffleBlockResolverSuite.java 1 addition, 1 deletion...rk/network/shuffle/ExternalShuffleBlockResolverSuite.java
- network/shuffle/src/test/java/org/apache/spark/network/shuffle/ExternalShuffleCleanupSuite.java 1 addition, 1 deletion...he/spark/network/shuffle/ExternalShuffleCleanupSuite.java
- network/shuffle/src/test/java/org/apache/spark/network/shuffle/ExternalShuffleIntegrationSuite.java 1 addition, 1 deletion...park/network/shuffle/ExternalShuffleIntegrationSuite.java
Loading
Please register or sign in to comment