-
- Downloads
[SPARK-21342] Fix DownloadCallback to work well with RetryingBlockFetcher.
## What changes were proposed in this pull request? When `RetryingBlockFetcher` retries fetching blocks. There could be two `DownloadCallback`s download the same content to the same target file. It could cause `ShuffleBlockFetcherIterator` reading a partial result. This pr proposes to create and delete the tmp files in `OneForOneBlockFetcher` Author: jinxing <jinxing6042@126.com> Author: Shixiong Zhu <zsxwing@gmail.com> Closes #18565 from jinxing64/SPARK-21342.
Showing
- common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalShuffleClient.java 3 additions, 4 deletions...g/apache/spark/network/shuffle/ExternalShuffleClient.java
- common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java 22 additions, 12 deletions...g/apache/spark/network/shuffle/OneForOneBlockFetcher.java
- common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ShuffleClient.java 11 additions, 2 deletions.../java/org/apache/spark/network/shuffle/ShuffleClient.java
- common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/TempShuffleFileManager.java 36 additions, 0 deletions.../apache/spark/network/shuffle/TempShuffleFileManager.java
- common/network-shuffle/src/test/java/org/apache/spark/network/sasl/SaslIntegrationSuite.java 1 addition, 1 deletion...a/org/apache/spark/network/sasl/SaslIntegrationSuite.java
- common/network-shuffle/src/test/java/org/apache/spark/network/shuffle/OneForOneBlockFetcherSuite.java 1 addition, 1 deletion...che/spark/network/shuffle/OneForOneBlockFetcherSuite.java
- core/src/main/scala/org/apache/spark/network/BlockTransferService.scala 4 additions, 4 deletions...scala/org/apache/spark/network/BlockTransferService.scala
- core/src/main/scala/org/apache/spark/network/netty/NettyBlockTransferService.scala 4 additions, 5 deletions...pache/spark/network/netty/NettyBlockTransferService.scala
- core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala 19 additions, 9 deletions...rg/apache/spark/storage/ShuffleBlockFetcherIterator.scala
- core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala 2 additions, 2 deletions...st/scala/org/apache/spark/storage/BlockManagerSuite.scala
- core/src/test/scala/org/apache/spark/storage/ShuffleBlockFetcherIteratorSuite.scala 5 additions, 5 deletions...ache/spark/storage/ShuffleBlockFetcherIteratorSuite.scala
Loading
Please register or sign in to comment