-
- Downloads
[SPARK-4452] [CORE] Shuffle data structures can starve others on the same thread for memory
## What changes were proposed in this pull request? In #9241 It implemented a mechanism to call spill() on those SQL operators that support spilling if there is not enough memory for execution. But ExternalSorter and AppendOnlyMap in Spark core are not worked. So this PR make them benefit from #9241. Now when there is not enough memory for execution, it can get memory by spilling ExternalSorter and AppendOnlyMap in Spark core. ## How was this patch tested? add two unit tests for it. Author: Lianhui Wang <lianhuiwang09@gmail.com> Closes #10024 from lianhuiwang/SPARK-4452-2.
Showing
- core/src/main/java/org/apache/spark/memory/MemoryConsumer.java 18 additions, 1 deletion...src/main/java/org/apache/spark/memory/MemoryConsumer.java
- core/src/main/java/org/apache/spark/memory/TaskMemoryManager.java 7 additions, 0 deletions.../main/java/org/apache/spark/memory/TaskMemoryManager.java
- core/src/main/scala/org/apache/spark/util/collection/ExternalAppendOnlyMap.scala 97 additions, 14 deletions.../apache/spark/util/collection/ExternalAppendOnlyMap.scala
- core/src/main/scala/org/apache/spark/util/collection/ExternalSorter.scala 132 additions, 18 deletions...ala/org/apache/spark/util/collection/ExternalSorter.scala
- core/src/main/scala/org/apache/spark/util/collection/Spillable.scala 35 additions, 13 deletions...in/scala/org/apache/spark/util/collection/Spillable.scala
- core/src/test/scala/org/apache/spark/util/collection/ExternalAppendOnlyMapSuite.scala 15 additions, 0 deletions...he/spark/util/collection/ExternalAppendOnlyMapSuite.scala
- core/src/test/scala/org/apache/spark/util/collection/ExternalSorterSuite.scala 17 additions, 0 deletions...rg/apache/spark/util/collection/ExternalSorterSuite.scala
- project/MimaExcludes.scala 3 additions, 0 deletionsproject/MimaExcludes.scala
Loading
Please register or sign in to comment