-
- Downloads
[SPARK-11017] [SQL] Support ImperativeAggregates in TungstenAggregate
This patch extends TungstenAggregate to support ImperativeAggregate functions. The existing TungstenAggregate operator only supported DeclarativeAggregate functions, which are defined in terms of Catalyst expressions and can be evaluated via generated projections. ImperativeAggregate functions, on the other hand, are evaluated by calling their `initialize`, `update`, `merge`, and `eval` methods. The basic strategy here is similar to how SortBasedAggregate evaluates both types of aggregate functions: use a generated projection to evaluate the expression-based declarative aggregates with dummy placeholder expressions inserted in place of the imperative aggregate function output, then invoke the imperative aggregate functions and target them against the aggregation buffer. The bulk of the diff here consists of code that was copied and adapted from SortBasedAggregate, with some key changes to handle TungstenAggregate's sort fallback path. Author: Josh Rosen <joshrosen@databricks.com> Closes #9038 from JoshRosen/support-interpreted-in-tungsten-agg-final.
Showing
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/functions.scala 17 additions, 2 deletions.../spark/sql/catalyst/expressions/aggregate/functions.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/interfaces.scala 20 additions, 11 deletions...spark/sql/catalyst/expressions/aggregate/interfaces.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/AggregationIterator.scala 15 additions, 14 deletions...e/spark/sql/execution/aggregate/AggregationIterator.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/TungstenAggregate.scala 21 additions, 1 deletion...che/spark/sql/execution/aggregate/TungstenAggregate.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/TungstenAggregationIterator.scala 199 additions, 51 deletions...sql/execution/aggregate/TungstenAggregationIterator.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/udaf.scala 37 additions, 42 deletions...scala/org/apache/spark/sql/execution/aggregate/udaf.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/utils.scala 134 additions, 135 deletions...cala/org/apache/spark/sql/execution/aggregate/utils.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/aggregate/TungstenAggregationIteratorSuite.scala 1 addition, 1 deletion...xecution/aggregate/TungstenAggregationIteratorSuite.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/hiveUDFs.scala 13 additions, 3 deletions...e/src/main/scala/org/apache/spark/sql/hive/hiveUDFs.scala
Loading
Please register or sign in to comment