-
- Downloads
[SPARK-9411] [SQL] Make Tungsten page sizes configurable
We need to make page sizes configurable so we can reduce them in unit tests and increase them in real production workloads. These sizes are now controlled by a new configuration, `spark.buffer.pageSize`. The new default is 64 megabytes. Author: Josh Rosen <joshrosen@databricks.com> Closes #7741 from JoshRosen/SPARK-9411 and squashes the following commits: a43c4db [Josh Rosen] Fix pow 2c0eefc [Josh Rosen] Fix MAXIMUM_PAGE_SIZE_BYTES comment + value bccfb51 [Josh Rosen] Lower page size to 4MB in TestHive ba54d4b [Josh Rosen] Make UnsafeExternalSorter's page size configurable 0045aa2 [Josh Rosen] Make UnsafeShuffle's page size configurable bc734f0 [Josh Rosen] Rename configuration e614858 [Josh Rosen] Makes BytesToBytesMap page size configurable
Showing
- core/src/main/java/org/apache/spark/shuffle/unsafe/UnsafeShuffleExternalSorter.java 21 additions, 14 deletions...che/spark/shuffle/unsafe/UnsafeShuffleExternalSorter.java
- core/src/main/java/org/apache/spark/shuffle/unsafe/UnsafeShuffleWriter.java 5 additions, 0 deletions.../org/apache/spark/shuffle/unsafe/UnsafeShuffleWriter.java
- core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java 16 additions, 14 deletions...ark/util/collection/unsafe/sort/UnsafeExternalSorter.java
- core/src/test/java/org/apache/spark/shuffle/unsafe/UnsafeShuffleWriterSuite.java 3 additions, 3 deletions...apache/spark/shuffle/unsafe/UnsafeShuffleWriterSuite.java
- sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeFixedWidthAggregationMap.java 4 additions, 1 deletion.../catalyst/expressions/UnsafeFixedWidthAggregationMap.java
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/UnsafeFixedWidthAggregationMapSuite.scala 5 additions, 1 deletion...yst/expressions/UnsafeFixedWidthAggregationMapSuite.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/GeneratedAggregate.scala 3 additions, 1 deletion...a/org/apache/spark/sql/execution/GeneratedAggregate.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala 6 additions, 1 deletion...org/apache/spark/sql/execution/joins/HashedRelation.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/test/TestHive.scala 1 addition, 0 deletions.../main/scala/org/apache/spark/sql/hive/test/TestHive.scala
- unsafe/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java 27 additions, 16 deletions...ain/java/org/apache/spark/unsafe/map/BytesToBytesMap.java
- unsafe/src/main/java/org/apache/spark/unsafe/memory/TaskMemoryManager.java 9 additions, 4 deletions...ava/org/apache/spark/unsafe/memory/TaskMemoryManager.java
- unsafe/src/test/java/org/apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java 12 additions, 10 deletions...apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java
Loading
Please register or sign in to comment