-
- Downloads
[SPARK-6717][ML] Clear shuffle files after checkpointing in ALS
## What changes were proposed in this pull request? When ALS is run with a checkpoint interval, during the checkpoint materialize the current state and cleanup the previous shuffles (non-blocking). ## How was this patch tested? Existing ALS unit tests, new ALS checkpoint cleanup unit tests added & shuffle files checked after ALS w/checkpointing run. Author: Holden Karau <holden@us.ibm.com> Author: Holden Karau <holden@pigscanfly.ca> Closes #11919 from holdenk/SPARK-6717-clear-shuffle-files-after-checkpointing-in-ALS.
Showing
- core/src/main/scala/org/apache/spark/ContextCleaner.scala 1 addition, 1 deletioncore/src/main/scala/org/apache/spark/ContextCleaner.scala
- mllib/src/main/scala/org/apache/spark/ml/recommendation/ALS.scala 33 additions, 2 deletions...c/main/scala/org/apache/spark/ml/recommendation/ALS.scala
- mllib/src/test/scala/org/apache/spark/ml/recommendation/ALSSuite.scala 161 additions, 42 deletions...t/scala/org/apache/spark/ml/recommendation/ALSSuite.scala
Loading
Please register or sign in to comment