-
- Downloads
[SPARK-14075] Refactor MemoryStore to be testable independent of BlockManager
This patch refactors the `MemoryStore` so that it can be tested without needing to construct / mock an entire `BlockManager`. - The block manager's serialization- and compression-related methods have been moved from `BlockManager` to `SerializerManager`. - `BlockInfoManager `is now passed directly to classes that need it, rather than being passed via the `BlockManager`. - The `MemoryStore` now calls `dropFromMemory` via a new `BlockEvictionHandler` interface rather than directly calling the `BlockManager`. This change helps to enforce a narrow interface between the `MemoryStore` and `BlockManager` functionality and makes this interface easier to mock in tests. - Several of the block unrolling tests have been moved from `BlockManagerSuite` into a new `MemoryStoreSuite`. Author: Josh Rosen <joshrosen@databricks.com> Closes #11899 from JoshRosen/reduce-memorystore-blockmanager-coupling.
Showing
- core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java 6 additions, 1 deletion...ain/java/org/apache/spark/unsafe/map/BytesToBytesMap.java
- core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java 12 additions, 5 deletions...ark/util/collection/unsafe/sort/UnsafeExternalSorter.java
- core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeSorterSpillReader.java 3 additions, 3 deletions.../util/collection/unsafe/sort/UnsafeSorterSpillReader.java
- core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeSorterSpillWriter.java 3 additions, 2 deletions.../util/collection/unsafe/sort/UnsafeSorterSpillWriter.java
- core/src/main/scala/org/apache/spark/serializer/SerializerManager.scala 88 additions, 2 deletions...scala/org/apache/spark/serializer/SerializerManager.scala
- core/src/main/scala/org/apache/spark/shuffle/BlockStoreShuffleReader.scala 3 additions, 2 deletions...la/org/apache/spark/shuffle/BlockStoreShuffleReader.scala
- core/src/main/scala/org/apache/spark/storage/BlockManager.scala 21 additions, 97 deletions...rc/main/scala/org/apache/spark/storage/BlockManager.scala
- core/src/main/scala/org/apache/spark/storage/BlockManagerManagedBuffer.scala 3 additions, 3 deletions.../org/apache/spark/storage/BlockManagerManagedBuffer.scala
- core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala 40 additions, 15 deletions...n/scala/org/apache/spark/storage/memory/MemoryStore.scala
- core/src/main/scala/org/apache/spark/util/collection/ExternalAppendOnlyMap.scala 4 additions, 3 deletions.../apache/spark/util/collection/ExternalAppendOnlyMap.scala
- core/src/main/scala/org/apache/spark/util/collection/ExternalSorter.scala 2 additions, 1 deletion...ala/org/apache/spark/util/collection/ExternalSorter.scala
- core/src/test/java/org/apache/spark/shuffle/sort/UnsafeShuffleWriterSuite.java 2 additions, 30 deletions...g/apache/spark/shuffle/sort/UnsafeShuffleWriterSuite.java
- core/src/test/java/org/apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java 10 additions, 7 deletions...apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java
- core/src/test/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorterSuite.java 8 additions, 4 deletions...til/collection/unsafe/sort/UnsafeExternalSorterSuite.java
- core/src/test/scala/org/apache/spark/DistributedSuite.scala 2 additions, 1 deletioncore/src/test/scala/org/apache/spark/DistributedSuite.scala
- core/src/test/scala/org/apache/spark/shuffle/BlockStoreShuffleReaderSuite.scala 8 additions, 14 deletions...g/apache/spark/shuffle/BlockStoreShuffleReaderSuite.scala
- core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala 1 addition, 196 deletions...st/scala/org/apache/spark/storage/BlockManagerSuite.scala
- core/src/test/scala/org/apache/spark/storage/MemoryStoreSuite.scala 302 additions, 0 deletions...est/scala/org/apache/spark/storage/MemoryStoreSuite.scala
- sql/catalyst/src/main/java/org/apache/spark/sql/execution/UnsafeExternalRowSorter.java 1 addition, 0 deletions...g/apache/spark/sql/execution/UnsafeExternalRowSorter.java
- sql/core/src/main/java/org/apache/spark/sql/execution/UnsafeFixedWidthAggregationMap.java 6 additions, 2 deletions...e/spark/sql/execution/UnsafeFixedWidthAggregationMap.java
Loading
Please register or sign in to comment