-
- Downloads
[SPARK-9824] [CORE] Fix the issue that InternalAccumulator leaks WeakReference
`InternalAccumulator.create` doesn't call `registerAccumulatorForCleanup` to register itself with ContextCleaner, so `WeakReference`s for these accumulators in `Accumulators.originals` won't be removed. This PR added `registerAccumulatorForCleanup` for internal accumulators to avoid the memory leak. Author: zsxwing <zsxwing@gmail.com> Closes #8108 from zsxwing/internal-accumulators-leak.
Showing
- core/src/main/scala/org/apache/spark/Accumulators.scala 13 additions, 9 deletionscore/src/main/scala/org/apache/spark/Accumulators.scala
- core/src/main/scala/org/apache/spark/scheduler/Stage.scala 1 addition, 1 deletioncore/src/main/scala/org/apache/spark/scheduler/Stage.scala
- core/src/test/scala/org/apache/spark/AccumulatorSuite.scala 2 additions, 1 deletioncore/src/test/scala/org/apache/spark/AccumulatorSuite.scala
Please register or sign in to comment