-
- Downloads
[SPARK-10984] Simplify *MemoryManager class structure
This patch refactors the MemoryManager class structure. After #9000, Spark had the following classes: - MemoryManager - StaticMemoryManager - ExecutorMemoryManager - TaskMemoryManager - ShuffleMemoryManager This is fairly confusing. To simplify things, this patch consolidates several of these classes: - ShuffleMemoryManager and ExecutorMemoryManager were merged into MemoryManager. - TaskMemoryManager is moved into Spark Core. **Key changes and tasks**: - [x] Merge ExecutorMemoryManager into MemoryManager. - [x] Move pooling logic into Allocator. - [x] Move TaskMemoryManager from `spark-unsafe` to `spark-core`. - [x] Refactor the existing Tungsten TaskMemoryManager interactions so Tungsten code use only this and not both this and ShuffleMemoryManager. - [x] Refactor non-Tungsten code to use the TaskMemoryManager instead of ShuffleMemoryManager. - [x] Merge ShuffleMemoryManager into MemoryManager. - [x] Move code - [x] ~~Simplify 1/n calculation.~~ **Will defer to followup, since this needs more work.** - [x] Port ShuffleMemoryManagerSuite tests. - [x] Move classes from `unsafe` package to `memory` package. - [ ] Figure out how to handle the hacky use of the memory managers in HashedRelation's broadcast variable construction. - [x] Test porting and cleanup: several tests relied on mock functionality (such as `TestShuffleMemoryManager.markAsOutOfMemory`) which has been changed or broken during the memory manager consolidation - [x] AbstractBytesToBytesMapSuite - [x] UnsafeExternalSorterSuite - [x] UnsafeFixedWidthAggregationMapSuite - [x] UnsafeKVExternalSorterSuite **Compatiblity notes**: - This patch introduces breaking changes in `ExternalAppendOnlyMap`, which is marked as `DevloperAPI` (likely for legacy reasons): this class now cannot be used outside of a task. Author: Josh Rosen <joshrosen@databricks.com> Closes #9127 from JoshRosen/SPARK-10984.
Showing
- core/src/main/java/org/apache/spark/memory/TaskMemoryManager.java 54 additions, 57 deletions.../main/java/org/apache/spark/memory/TaskMemoryManager.java
- core/src/main/java/org/apache/spark/shuffle/sort/PackedRecordPointer.java 3 additions, 1 deletion...va/org/apache/spark/shuffle/sort/PackedRecordPointer.java
- core/src/main/java/org/apache/spark/shuffle/sort/ShuffleExternalSorter.java 26 additions, 31 deletions.../org/apache/spark/shuffle/sort/ShuffleExternalSorter.java
- core/src/main/java/org/apache/spark/shuffle/sort/UnsafeShuffleWriter.java 1 addition, 6 deletions...va/org/apache/spark/shuffle/sort/UnsafeShuffleWriter.java
- core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java 9 additions, 27 deletions...ain/java/org/apache/spark/unsafe/map/BytesToBytesMap.java
- core/src/main/java/org/apache/spark/util/collection/unsafe/sort/RecordPointerAndKeyPrefix.java 3 additions, 1 deletion...til/collection/unsafe/sort/RecordPointerAndKeyPrefix.java
- core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java 17 additions, 34 deletions...ark/util/collection/unsafe/sort/UnsafeExternalSorter.java
- core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeInMemorySorter.java 1 addition, 1 deletion...ark/util/collection/unsafe/sort/UnsafeInMemorySorter.java
- core/src/main/scala/org/apache/spark/SparkEnv.scala 3 additions, 20 deletionscore/src/main/scala/org/apache/spark/SparkEnv.scala
- core/src/main/scala/org/apache/spark/TaskContext.scala 1 addition, 1 deletioncore/src/main/scala/org/apache/spark/TaskContext.scala
- core/src/main/scala/org/apache/spark/TaskContextImpl.scala 1 addition, 1 deletioncore/src/main/scala/org/apache/spark/TaskContextImpl.scala
- core/src/main/scala/org/apache/spark/executor/Executor.scala 2 additions, 2 deletionscore/src/main/scala/org/apache/spark/executor/Executor.scala
- core/src/main/scala/org/apache/spark/memory/MemoryManager.scala 180 additions, 17 deletions...rc/main/scala/org/apache/spark/memory/MemoryManager.scala
- core/src/main/scala/org/apache/spark/memory/StaticMemoryManager.scala 7 additions, 5 deletions...n/scala/org/apache/spark/memory/StaticMemoryManager.scala
- core/src/main/scala/org/apache/spark/memory/UnifiedMemoryManager.scala 8 additions, 4 deletions.../scala/org/apache/spark/memory/UnifiedMemoryManager.scala
- core/src/main/scala/org/apache/spark/scheduler/Task.scala 1 addition, 5 deletionscore/src/main/scala/org/apache/spark/scheduler/Task.scala
- core/src/main/scala/org/apache/spark/shuffle/BlockStoreShuffleReader.scala 3 additions, 2 deletions...la/org/apache/spark/shuffle/BlockStoreShuffleReader.scala
- core/src/main/scala/org/apache/spark/shuffle/ShuffleMemoryManager.scala 0 additions, 209 deletions...scala/org/apache/spark/shuffle/ShuffleMemoryManager.scala
- core/src/main/scala/org/apache/spark/shuffle/sort/SortShuffleManager.scala 0 additions, 1 deletion...la/org/apache/spark/shuffle/sort/SortShuffleManager.scala
- core/src/main/scala/org/apache/spark/shuffle/sort/SortShuffleWriter.scala 3 additions, 3 deletions...ala/org/apache/spark/shuffle/sort/SortShuffleWriter.scala
Loading
Please register or sign in to comment