-
- Downloads
[SPARK-6578] [core] Fix thread-safety issue in outbound path of network library.
While the inbound path of a netty pipeline is thread-safe, the outbound path is not. That means that multiple threads can compete to write messages to the next stage of the pipeline. The network library sometimes breaks a single RPC message into multiple buffers internally to avoid copying data (see MessageEncoder). This can result in the following scenario (where "FxBy" means "frame x, buffer y"): T1 F1B1 F1B2 \ \ \ \ socket F1B1 F2B1 F1B2 F2B2 / / / / T2 F2B1 F2B2 And the frames now cannot be rebuilt on the receiving side because the different messages have been mixed up on the wire. The fix wraps these multi-buffer messages into a `FileRegion` object so that these messages are written "atomically" to the next pipeline handler. Author: Marcelo Vanzin <vanzin@cloudera.com> Closes #5234 from vanzin/SPARK-6578 and squashes the following commits: 16b2d70 [Marcelo Vanzin] Forgot to update a type. c9c2e4e [Marcelo Vanzin] Review comments: simplify some code. 9c888ac [Marcelo Vanzin] Small style nits. 8474bab [Marcelo Vanzin] Fix multiple calls to MessageWithHeader.transferTo(). e26509f [Marcelo Vanzin] Merge branch 'master' into SPARK-6578 c503f6c [Marcelo Vanzin] Implement a custom FileRegion instead of using locks. 84aa7ce [Marcelo Vanzin] Rename handler to the correct name. 432f3bd [Marcelo Vanzin] Remove unneeded method. 8d70e60 [Marcelo Vanzin] Fix thread-safety issue in outbound path of network library.
Showing
- network/common/pom.xml 5 additions, 0 deletionsnetwork/common/pom.xml
- network/common/src/main/java/org/apache/spark/network/protocol/MessageEncoder.java 4 additions, 2 deletions...ava/org/apache/spark/network/protocol/MessageEncoder.java
- network/common/src/main/java/org/apache/spark/network/protocol/MessageWithHeader.java 106 additions, 0 deletions.../org/apache/spark/network/protocol/MessageWithHeader.java
- network/common/src/test/java/org/apache/spark/network/ByteArrayWritableChannel.java 55 additions, 0 deletions...va/org/apache/spark/network/ByteArrayWritableChannel.java
- network/common/src/test/java/org/apache/spark/network/ProtocolSuite.java 38 additions, 8 deletions...src/test/java/org/apache/spark/network/ProtocolSuite.java
- network/common/src/test/java/org/apache/spark/network/protocol/MessageWithHeaderSuite.java 129 additions, 0 deletions...apache/spark/network/protocol/MessageWithHeaderSuite.java
- network/common/src/test/resources/log4j.properties 27 additions, 0 deletionsnetwork/common/src/test/resources/log4j.properties
Loading
Please register or sign in to comment