-
- Downloads
[SPARK-20994] Remove redundant characters in OpenBlocks to save memory for shuffle service.
## What changes were proposed in this pull request? In current code, blockIds in `OpenBlocks` are stored in the iterator on shuffle service. There are some redundant characters in blockId(`"shuffle_" + shuffleId + "_" + mapId + "_" + reduceId`). This pr proposes to improve the footprint and alleviate the memory pressure on shuffle service. Author: jinxing <jinxing6042@126.com> Closes #18231 from jinxing64/SPARK-20994-v2.
Showing
- common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalShuffleBlockHandler.java 49 additions, 21 deletions...he/spark/network/shuffle/ExternalShuffleBlockHandler.java
- common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalShuffleBlockResolver.java 8 additions, 15 deletions...e/spark/network/shuffle/ExternalShuffleBlockResolver.java
- common/network-shuffle/src/test/java/org/apache/spark/network/sasl/SaslIntegrationSuite.java 1 addition, 1 deletion...a/org/apache/spark/network/sasl/SaslIntegrationSuite.java
- common/network-shuffle/src/test/java/org/apache/spark/network/shuffle/ExternalShuffleBlockHandlerSuite.java 6 additions, 5 deletions...ark/network/shuffle/ExternalShuffleBlockHandlerSuite.java
- common/network-shuffle/src/test/java/org/apache/spark/network/shuffle/ExternalShuffleBlockResolverSuite.java 5 additions, 5 deletions...rk/network/shuffle/ExternalShuffleBlockResolverSuite.java
- common/network-shuffle/src/test/java/org/apache/spark/network/shuffle/ExternalShuffleIntegrationSuite.java 4 additions, 4 deletions...park/network/shuffle/ExternalShuffleIntegrationSuite.java
Loading
Please register or sign in to comment