-
- Downloads
[SPARK-11425] [SPARK-11486] Improve hybrid aggregation
After aggregation, the dataset could be smaller than inputs, so it's better to do hash based aggregation for all inputs, then using sort based aggregation to merge them. Author: Davies Liu <davies@databricks.com> Closes #9383 from davies/fix_switch.
Showing
- core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java 33 additions, 13 deletions...ain/java/org/apache/spark/unsafe/map/BytesToBytesMap.java
- core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java 26 additions, 13 deletions...ark/util/collection/unsafe/sort/UnsafeExternalSorter.java
- core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeInMemorySorter.java 11 additions, 4 deletions...ark/util/collection/unsafe/sort/UnsafeInMemorySorter.java
- sql/core/src/main/java/org/apache/spark/sql/execution/UnsafeFixedWidthAggregationMap.java 3 additions, 6 deletions...e/spark/sql/execution/UnsafeFixedWidthAggregationMap.java
- sql/core/src/main/java/org/apache/spark/sql/execution/UnsafeKVExternalSorter.java 15 additions, 8 deletions...rg/apache/spark/sql/execution/UnsafeKVExternalSorter.java
- sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/TungstenAggregationIterator.scala 42 additions, 129 deletions...sql/execution/aggregate/TungstenAggregationIterator.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/UnsafeFixedWidthAggregationMapSuite.scala 26 additions, 38 deletions...k/sql/execution/UnsafeFixedWidthAggregationMapSuite.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/AggregationQuerySuite.scala 9 additions, 3 deletions...ache/spark/sql/hive/execution/AggregationQuerySuite.scala
Loading
Please register or sign in to comment