-
- Downloads
[SPARK-7542][SQL] Support off-heap index/sort buffer
This brings the support of off-heap memory for array inside BytesToBytesMap and InMemorySorter, then we could allocate all the memory from off-heap for execution. Closes #8068 Author: Davies Liu <davies@databricks.com> Closes #9477 from davies/unsafe_timsort.
Showing
- core/src/main/java/org/apache/spark/memory/MemoryConsumer.java 20 additions, 16 deletions...src/main/java/org/apache/spark/memory/MemoryConsumer.java
- core/src/main/java/org/apache/spark/memory/TaskMemoryManager.java 4 additions, 2 deletions.../main/java/org/apache/spark/memory/TaskMemoryManager.java
- core/src/main/java/org/apache/spark/shuffle/sort/ShuffleExternalSorter.java 9 additions, 17 deletions.../org/apache/spark/shuffle/sort/ShuffleExternalSorter.java
- core/src/main/java/org/apache/spark/shuffle/sort/ShuffleInMemorySorter.java 35 additions, 32 deletions.../org/apache/spark/shuffle/sort/ShuffleInMemorySorter.java
- core/src/main/java/org/apache/spark/shuffle/sort/ShuffleSortDataFormat.java 24 additions, 14 deletions.../org/apache/spark/shuffle/sort/ShuffleSortDataFormat.java
- core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java 8 additions, 10 deletions...ain/java/org/apache/spark/unsafe/map/BytesToBytesMap.java
- core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java 10 additions, 18 deletions...ark/util/collection/unsafe/sort/UnsafeExternalSorter.java
- core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeInMemorySorter.java 40 additions, 26 deletions...ark/util/collection/unsafe/sort/UnsafeInMemorySorter.java
- core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeSortDataFormat.java 28 additions, 19 deletions...ark/util/collection/unsafe/sort/UnsafeSortDataFormat.java
- core/src/test/java/org/apache/spark/memory/TaskMemoryManagerSuite.java 0 additions, 23 deletions.../java/org/apache/spark/memory/TaskMemoryManagerSuite.java
- core/src/test/java/org/apache/spark/memory/TestMemoryConsumer.java 45 additions, 0 deletions...test/java/org/apache/spark/memory/TestMemoryConsumer.java
- core/src/test/java/org/apache/spark/shuffle/sort/ShuffleInMemorySorterSuite.java 11 additions, 5 deletions...apache/spark/shuffle/sort/ShuffleInMemorySorterSuite.java
- core/src/test/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorterSuite.java 0 additions, 1 deletion...til/collection/unsafe/sort/UnsafeExternalSorterSuite.java
- core/src/test/java/org/apache/spark/util/collection/unsafe/sort/UnsafeInMemorySorterSuite.java 8 additions, 4 deletions...til/collection/unsafe/sort/UnsafeInMemorySorterSuite.java
- sql/core/src/main/java/org/apache/spark/sql/execution/UnsafeKVExternalSorter.java 2 additions, 1 deletion...rg/apache/spark/sql/execution/UnsafeKVExternalSorter.java
- unsafe/src/main/java/org/apache/spark/unsafe/array/LongArray.java 17 additions, 1 deletion...rc/main/java/org/apache/spark/unsafe/array/LongArray.java
- unsafe/src/test/java/org/apache/spark/unsafe/array/LongArraySuite.java 4 additions, 0 deletions...st/java/org/apache/spark/unsafe/array/LongArraySuite.java
Loading
Please register or sign in to comment