-
- Downloads
[SPARK-10997][CORE] Add "client mode" to netty rpc env.
"Client mode" means the RPC env will not listen for incoming connections. This allows certain processes in the Spark stack (such as Executors or tha YARN client-mode AM) to act as pure clients when using the netty-based RPC backend, reducing the number of sockets needed by the app and also the number of open ports. Client connections are also preferred when endpoints that actually have a listening socket are involved; so, for example, if a Worker connects to a Master and the Master needs to send a message to a Worker endpoint, that client connection will be used, even though the Worker is also listening for incoming connections. With this change, the workaround for SPARK-10987 isn't necessary anymore, and is removed. The AM connects to the driver in "client mode", and that connection is used for all driver <-> AM communication, and so the AM is properly notified when the connection goes down. Author: Marcelo Vanzin <vanzin@cloudera.com> Closes #9210 from vanzin/SPARK-10997.
Showing
- core/src/main/scala/org/apache/spark/SparkEnv.scala 5 additions, 2 deletionscore/src/main/scala/org/apache/spark/SparkEnv.scala
- core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala 9 additions, 11 deletions.../apache/spark/executor/CoarseGrainedExecutorBackend.scala
- core/src/main/scala/org/apache/spark/rpc/RpcEnv.scala 5 additions, 3 deletionscore/src/main/scala/org/apache/spark/rpc/RpcEnv.scala
- core/src/main/scala/org/apache/spark/rpc/netty/Dispatcher.scala 1 addition, 1 deletion...rc/main/scala/org/apache/spark/rpc/netty/Dispatcher.scala
- core/src/main/scala/org/apache/spark/rpc/netty/NettyRpcEnv.scala 133 additions, 112 deletions...c/main/scala/org/apache/spark/rpc/netty/NettyRpcEnv.scala
- core/src/main/scala/org/apache/spark/rpc/netty/Outbox.scala 19 additions, 5 deletionscore/src/main/scala/org/apache/spark/rpc/netty/Outbox.scala
- core/src/main/scala/org/apache/spark/rpc/netty/RpcEndpointAddress.scala 17 additions, 7 deletions...scala/org/apache/spark/rpc/netty/RpcEndpointAddress.scala
- core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedClusterMessage.scala 2 additions, 8 deletions...spark/scheduler/cluster/CoarseGrainedClusterMessage.scala
- core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala 12 additions, 6 deletions...ark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
- core/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala 0 additions, 2 deletions...apache/spark/scheduler/cluster/YarnSchedulerBackend.scala
- core/src/test/scala/org/apache/spark/rpc/RpcEnvSuite.scala 32 additions, 18 deletionscore/src/test/scala/org/apache/spark/rpc/RpcEnvSuite.scala
- core/src/test/scala/org/apache/spark/rpc/akka/AkkaRpcEnvSuite.scala 7 additions, 4 deletions...est/scala/org/apache/spark/rpc/akka/AkkaRpcEnvSuite.scala
- core/src/test/scala/org/apache/spark/rpc/netty/NettyRpcAddressSuite.scala 6 additions, 1 deletion...ala/org/apache/spark/rpc/netty/NettyRpcAddressSuite.scala
- core/src/test/scala/org/apache/spark/rpc/netty/NettyRpcEnvSuite.scala 7 additions, 2 deletions...t/scala/org/apache/spark/rpc/netty/NettyRpcEnvSuite.scala
- core/src/test/scala/org/apache/spark/rpc/netty/NettyRpcHandlerSuite.scala 4 additions, 4 deletions...ala/org/apache/spark/rpc/netty/NettyRpcHandlerSuite.scala
- network/yarn/pom.xml 5 additions, 0 deletionsnetwork/yarn/pom.xml
- yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala 2 additions, 4 deletions...cala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
Loading
Please register or sign in to comment