-
- Downloads
[SPARK-15357] Cooperative spilling should check consumer memory mode
## What changes were proposed in this pull request? Since we support forced spilling for Spillable, which only works in OnHeap mode, different from other SQL operators (could be OnHeap or OffHeap), we should considering the mode of consumer before calling trigger forced spilling. ## How was this patch tested? Add new test. Author: Davies Liu <davies@databricks.com> Closes #13151 from davies/fix_mode.
Showing
- core/src/main/java/org/apache/spark/memory/MemoryConsumer.java 17 additions, 8 deletions...src/main/java/org/apache/spark/memory/MemoryConsumer.java
- core/src/main/java/org/apache/spark/memory/TaskMemoryManager.java 16 additions, 21 deletions.../main/java/org/apache/spark/memory/TaskMemoryManager.java
- core/src/main/java/org/apache/spark/shuffle/sort/ShuffleExternalSorter.java 3 additions, 2 deletions.../org/apache/spark/shuffle/sort/ShuffleExternalSorter.java
- core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java 1 addition, 1 deletion...ain/java/org/apache/spark/unsafe/map/BytesToBytesMap.java
- core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java 1 addition, 1 deletion...ark/util/collection/unsafe/sort/UnsafeExternalSorter.java
- core/src/main/scala/org/apache/spark/executor/Executor.scala 5 additions, 5 deletionscore/src/main/scala/org/apache/spark/executor/Executor.scala
- core/src/main/scala/org/apache/spark/util/collection/Spillable.scala 2 additions, 2 deletions...in/scala/org/apache/spark/util/collection/Spillable.scala
- core/src/test/java/org/apache/spark/memory/TaskMemoryManagerSuite.java 25 additions, 3 deletions.../java/org/apache/spark/memory/TaskMemoryManagerSuite.java
- core/src/test/java/org/apache/spark/memory/TestMemoryConsumer.java 6 additions, 9 deletions...test/java/org/apache/spark/memory/TestMemoryConsumer.java
- core/src/test/java/org/apache/spark/shuffle/sort/PackedRecordPointerSuite.java 7 additions, 6 deletions...g/apache/spark/shuffle/sort/PackedRecordPointerSuite.java
- core/src/test/java/org/apache/spark/shuffle/sort/ShuffleInMemorySorterSuite.java 3 additions, 1 deletion...apache/spark/shuffle/sort/ShuffleInMemorySorterSuite.java
- core/src/test/java/org/apache/spark/util/collection/unsafe/sort/UnsafeInMemorySorterSuite.java 1 addition, 1 deletion...til/collection/unsafe/sort/UnsafeInMemorySorterSuite.java
- core/src/test/scala/org/apache/spark/FailureSuite.scala 5 additions, 2 deletionscore/src/test/scala/org/apache/spark/FailureSuite.scala
- core/src/test/scala/org/apache/spark/memory/MemoryManagerSuite.scala 46 additions, 34 deletions...st/scala/org/apache/spark/memory/MemoryManagerSuite.scala
- project/MimaExcludes.scala 1 addition, 0 deletionsproject/MimaExcludes.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala 6 additions, 10 deletions...org/apache/spark/sql/execution/joins/HashedRelation.scala
Loading
Please register or sign in to comment