-
- Downloads
[SPARK-11866][NETWORK][CORE] Make sure timed out RPCs are cleaned up.
This change does a couple of different things to make sure that the RpcEnv-level code and the network library agree about the status of outstanding RPCs. For RPCs that do not expect a reply ("RpcEnv.send"), support for one way messages (hello CORBA!) was added to the network layer. This is a "fire and forget" message that does not require any state to be kept by the TransportClient; as a result, the RpcEnv 'Ack' message is not needed anymore. For RPCs that do expect a reply ("RpcEnv.ask"), the network library now returns the internal RPC id; if the RpcEnv layer decides to time out the RPC before the network layer does, it now asks the TransportClient to forget about the RPC, so that if the network-level timeout occurs, the client is not killed. As part of implementing the above, I cleaned up some of the code in the netty rpc backend, removing types that were not necessary and factoring out some common code. Of interest is a slight change in the exceptions when posting messages to a stopped RpcEnv; that's mostly to avoid nasty error messages from the local-cluster backend when shutting down, which pollutes the terminal output. Author: Marcelo Vanzin <vanzin@cloudera.com> Closes #9917 from vanzin/SPARK-11866.
Showing
- core/src/main/scala/org/apache/spark/deploy/worker/ExecutorRunner.scala 5 additions, 1 deletion...scala/org/apache/spark/deploy/worker/ExecutorRunner.scala
- core/src/main/scala/org/apache/spark/rpc/netty/Dispatcher.scala 23 additions, 32 deletions...rc/main/scala/org/apache/spark/rpc/netty/Dispatcher.scala
- core/src/main/scala/org/apache/spark/rpc/netty/Inbox.scala 13 additions, 15 deletionscore/src/main/scala/org/apache/spark/rpc/netty/Inbox.scala
- core/src/main/scala/org/apache/spark/rpc/netty/NettyRpcCallContext.scala 6 additions, 29 deletions...cala/org/apache/spark/rpc/netty/NettyRpcCallContext.scala
- core/src/main/scala/org/apache/spark/rpc/netty/NettyRpcEnv.scala 62 additions, 91 deletions...c/main/scala/org/apache/spark/rpc/netty/NettyRpcEnv.scala
- core/src/main/scala/org/apache/spark/rpc/netty/Outbox.scala 49 additions, 15 deletionscore/src/main/scala/org/apache/spark/rpc/netty/Outbox.scala
- core/src/test/scala/org/apache/spark/rpc/netty/InboxSuite.scala 3 additions, 3 deletions...rc/test/scala/org/apache/spark/rpc/netty/InboxSuite.scala
- core/src/test/scala/org/apache/spark/rpc/netty/NettyRpcHandlerSuite.scala 1 addition, 1 deletion...ala/org/apache/spark/rpc/netty/NettyRpcHandlerSuite.scala
- network/common/src/main/java/org/apache/spark/network/client/TransportClient.java 33 additions, 1 deletion...java/org/apache/spark/network/client/TransportClient.java
- network/common/src/main/java/org/apache/spark/network/protocol/Message.java 3 additions, 1 deletion.../main/java/org/apache/spark/network/protocol/Message.java
- network/common/src/main/java/org/apache/spark/network/protocol/MessageDecoder.java 3 additions, 0 deletions...ava/org/apache/spark/network/protocol/MessageDecoder.java
- network/common/src/main/java/org/apache/spark/network/protocol/OneWayMessage.java 75 additions, 0 deletions...java/org/apache/spark/network/protocol/OneWayMessage.java
- network/common/src/main/java/org/apache/spark/network/sasl/SaslRpcHandler.java 5 additions, 0 deletions...in/java/org/apache/spark/network/sasl/SaslRpcHandler.java
- network/common/src/main/java/org/apache/spark/network/server/RpcHandler.java 36 additions, 0 deletions...main/java/org/apache/spark/network/server/RpcHandler.java
- network/common/src/main/java/org/apache/spark/network/server/TransportRequestHandler.java 15 additions, 3 deletions.../apache/spark/network/server/TransportRequestHandler.java
- network/common/src/test/java/org/apache/spark/network/ProtocolSuite.java 2 additions, 0 deletions...src/test/java/org/apache/spark/network/ProtocolSuite.java
- network/common/src/test/java/org/apache/spark/network/RpcIntegrationSuite.java 31 additions, 0 deletions...st/java/org/apache/spark/network/RpcIntegrationSuite.java
- network/common/src/test/java/org/apache/spark/network/sasl/SparkSaslSuite.java 9 additions, 0 deletions...st/java/org/apache/spark/network/sasl/SparkSaslSuite.java
Loading
Please register or sign in to comment