-
- Downloads
[SPARK-9747] [SQL] Avoid starving an unsafe operator in aggregation
This is the sister patch to #8011, but for aggregation. In a nutshell: create the `TungstenAggregationIterator` before computing the parent partition. Internally this creates a `BytesToBytesMap` which acquires a page in the constructor as of this patch. This ensures that the aggregation operator is not starved since we reserve at least 1 page in advance. rxin yhuai Author: Andrew Or <andrew@databricks.com> Closes #8038 from andrewor14/unsafe-starve-memory-agg.
Showing
- core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java 25 additions, 9 deletions...ain/java/org/apache/spark/unsafe/map/BytesToBytesMap.java
- core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java 4 additions, 5 deletions...ark/util/collection/unsafe/sort/UnsafeExternalSorter.java
- core/src/test/java/org/apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java 10 additions, 1 deletion...apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java
- sql/core/src/main/java/org/apache/spark/sql/execution/UnsafeFixedWidthAggregationMap.java 7 additions, 0 deletions...e/spark/sql/execution/UnsafeFixedWidthAggregationMap.java
- sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/TungstenAggregate.scala 47 additions, 25 deletions...che/spark/sql/execution/aggregate/TungstenAggregate.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/TungstenAggregationIterator.scala 52 additions, 36 deletions...sql/execution/aggregate/TungstenAggregationIterator.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/aggregate/TungstenAggregationIteratorSuite.scala 56 additions, 0 deletions...xecution/aggregate/TungstenAggregationIteratorSuite.scala
Loading
Please register or sign in to comment