-
- Downloads
[SPARK-4236] Cleanup removed applications' files in shuffle service
This relies on a hook from whoever is hosting the shuffle service to invoke removeApplication() when the application is completed. Once invoked, we will clean up all the executors' shuffle directories we know about. Author: Aaron Davidson <aaron@databricks.com> Closes #3126 from aarondav/cleanup and squashes the following commits: 33a64a9 [Aaron Davidson] Missing brace e6e428f [Aaron Davidson] Address comments 16a0d27 [Aaron Davidson] Cleanup e4df3e7 [Aaron Davidson] [SPARK-4236] Cleanup removed applications' files in shuffle service
Showing
- core/src/main/scala/org/apache/spark/util/Utils.scala 1 addition, 0 deletionscore/src/main/scala/org/apache/spark/util/Utils.scala
- core/src/test/scala/org/apache/spark/ExternalShuffleServiceSuite.scala 3 additions, 2 deletions.../scala/org/apache/spark/ExternalShuffleServiceSuite.scala
- network/common/src/main/java/org/apache/spark/network/util/JavaUtils.java 59 additions, 0 deletions...rc/main/java/org/apache/spark/network/util/JavaUtils.java
- network/shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalShuffleBlockHandler.java 6 additions, 4 deletions...he/spark/network/shuffle/ExternalShuffleBlockHandler.java
- network/shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalShuffleBlockManager.java 105 additions, 13 deletions...he/spark/network/shuffle/ExternalShuffleBlockManager.java
- network/shuffle/src/test/java/org/apache/spark/network/shuffle/ExternalShuffleCleanupSuite.java 142 additions, 0 deletions...he/spark/network/shuffle/ExternalShuffleCleanupSuite.java
- network/shuffle/src/test/java/org/apache/spark/network/shuffle/ExternalShuffleIntegrationSuite.java 1 addition, 1 deletion...park/network/shuffle/ExternalShuffleIntegrationSuite.java
- network/shuffle/src/test/java/org/apache/spark/network/shuffle/TestShuffleDataContext.java 2 additions, 2 deletions.../apache/spark/network/shuffle/TestShuffleDataContext.java
Loading
Please register or sign in to comment