-
- Downloads
[SPARK-3019] Pluggable block transfer interface (BlockTransferService)
This pull request creates a new BlockTransferService interface for block fetch/upload and refactors the existing ConnectionManager to implement BlockTransferService (NioBlockTransferService). Most of the changes are simply moving code around. The main class to inspect is ShuffleBlockFetcherIterator. Review guide: - Most of the ConnectionManager code is now in network.cm package - ManagedBuffer is a new buffer abstraction backed by several different implementations (file segment, nio ByteBuffer, Netty ByteBuf) - BlockTransferService is the main internal interface introduced in this PR - NioBlockTransferService implements BlockTransferService and replaces the old BlockManagerWorker - ShuffleBlockFetcherIterator replaces the told BlockFetcherIterator to use the new interface TODOs that should be separate PRs: - Implement NettyBlockTransferService - Finalize the API/semantics for ManagedBuffer.release() Author: Reynold Xin <rxin@apache.org> Closes #2240 from rxin/blockTransferService and squashes the following commits: 64cd9d7 [Reynold Xin] Merge branch 'master' into blockTransferService 1dfd3d7 [Reynold Xin] Limit the length of the FileInputStream. 1332156 [Reynold Xin] Fixed style violation from refactoring. 2960c93 [Reynold Xin] Added ShuffleBlockFetcherIteratorSuite. e29c721 [Reynold Xin] Updated comment for ShuffleBlockFetcherIterator. 8a1046e [Reynold Xin] Code review feedback: 2c6b1e1 [Reynold Xin] Removed println in test cases. 2a907e4 [Reynold Xin] Merge branch 'master' into blockTransferService-merge 07ccf0d [Reynold Xin] Added init check to CMBlockTransferService. 98c668a [Reynold Xin] Added failure handling and fixed unit tests. ae05fcd [Reynold Xin] Updated tests, although DistributedSuite is hanging. d8d595c [Reynold Xin] Merge branch 'master' of github.com:apache/spark into blockTransferService 9ef279c [Reynold Xin] Initial refactoring to move ConnectionManager to use the BlockTransferService.
Showing
- core/src/main/scala/org/apache/spark/SparkEnv.scala 9 additions, 6 deletionscore/src/main/scala/org/apache/spark/SparkEnv.scala
- core/src/main/scala/org/apache/spark/network/BlockDataManager.scala 14 additions, 15 deletions...ain/scala/org/apache/spark/network/BlockDataManager.scala
- core/src/main/scala/org/apache/spark/network/BlockFetchingListener.scala 37 additions, 0 deletions...cala/org/apache/spark/network/BlockFetchingListener.scala
- core/src/main/scala/org/apache/spark/network/BlockTransferService.scala 131 additions, 0 deletions...scala/org/apache/spark/network/BlockTransferService.scala
- core/src/main/scala/org/apache/spark/network/ConnectionManagerTest.scala 0 additions, 103 deletions...cala/org/apache/spark/network/ConnectionManagerTest.scala
- core/src/main/scala/org/apache/spark/network/ManagedBuffer.scala 107 additions, 0 deletions...c/main/scala/org/apache/spark/network/ManagedBuffer.scala
- core/src/main/scala/org/apache/spark/network/SenderTest.scala 0 additions, 76 deletions.../src/main/scala/org/apache/spark/network/SenderTest.scala
- core/src/main/scala/org/apache/spark/network/nio/BlockMessage.scala 6 additions, 18 deletions...ain/scala/org/apache/spark/network/nio/BlockMessage.scala
- core/src/main/scala/org/apache/spark/network/nio/BlockMessageArray.scala 6 additions, 6 deletions...cala/org/apache/spark/network/nio/BlockMessageArray.scala
- core/src/main/scala/org/apache/spark/network/nio/BufferMessage.scala 3 additions, 2 deletions...in/scala/org/apache/spark/network/nio/BufferMessage.scala
- core/src/main/scala/org/apache/spark/network/nio/Connection.scala 5 additions, 5 deletions.../main/scala/org/apache/spark/network/nio/Connection.scala
- core/src/main/scala/org/apache/spark/network/nio/ConnectionId.scala 3 additions, 3 deletions...ain/scala/org/apache/spark/network/nio/ConnectionId.scala
- core/src/main/scala/org/apache/spark/network/nio/ConnectionManager.scala 9 additions, 14 deletions...cala/org/apache/spark/network/nio/ConnectionManager.scala
- core/src/main/scala/org/apache/spark/network/nio/ConnectionManagerId.scala 3 additions, 3 deletions...la/org/apache/spark/network/nio/ConnectionManagerId.scala
- core/src/main/scala/org/apache/spark/network/nio/Message.scala 4 additions, 3 deletions...src/main/scala/org/apache/spark/network/nio/Message.scala
- core/src/main/scala/org/apache/spark/network/nio/MessageChunk.scala 2 additions, 2 deletions...ain/scala/org/apache/spark/network/nio/MessageChunk.scala
- core/src/main/scala/org/apache/spark/network/nio/MessageChunkHeader.scala 4 additions, 5 deletions...ala/org/apache/spark/network/nio/MessageChunkHeader.scala
- core/src/main/scala/org/apache/spark/network/nio/NioBlockTransferService.scala 205 additions, 0 deletions...rg/apache/spark/network/nio/NioBlockTransferService.scala
- core/src/main/scala/org/apache/spark/network/nio/SecurityMessage.scala 4 additions, 6 deletions.../scala/org/apache/spark/network/nio/SecurityMessage.scala
- core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala 1 addition, 1 deletion...in/scala/org/apache/spark/serializer/KryoSerializer.scala
Loading
Please register or sign in to comment