-
- Downloads
[SPARK-14275][SQL] Reimplement TypedAggregateExpression to DeclarativeAggregate
## What changes were proposed in this pull request? `ExpressionEncoder` is just a container for serialization and deserialization expressions, we can use these expressions to build `TypedAggregateExpression` directly, so that it can fit in `DeclarativeAggregate`, which is more efficient. One trick is, for each buffer serializer expression, it will reference to the result object of serialization and function call. To avoid re-calculating this result object, we can serialize the buffer object to a single struct field, so that we can use a special `Expression` to only evaluate result object once. ## How was this patch tested? existing tests Author: Wenchen Fan <wenchen@databricks.com> Closes #12067 from cloud-fan/typed_udaf.
Showing
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala 1 addition, 1 deletion...rg/apache/spark/sql/catalyst/expressions/Expression.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ReferenceToExpressions.scala 77 additions, 0 deletions...ark/sql/catalyst/expressions/ReferenceToExpressions.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/literals.scala 2 additions, 1 deletion.../org/apache/spark/sql/catalyst/expressions/literals.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/types/ObjectType.scala 2 additions, 0 deletions...rc/main/scala/org/apache/spark/sql/types/ObjectType.scala
- sql/core/src/main/scala/org/apache/spark/sql/Column.scala 8 additions, 8 deletionssql/core/src/main/scala/org/apache/spark/sql/Column.scala
- sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala 2 additions, 2 deletionssql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
- sql/core/src/main/scala/org/apache/spark/sql/KeyValueGroupedDataset.scala 1 addition, 2 deletions...n/scala/org/apache/spark/sql/KeyValueGroupedDataset.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/TypedAggregateExpression.scala 106 additions, 86 deletions...rk/sql/execution/aggregate/TypedAggregateExpression.scala
- sql/core/src/main/scala/org/apache/spark/sql/expressions/Aggregator.scala 3 additions, 3 deletions...n/scala/org/apache/spark/sql/expressions/Aggregator.scala
- sql/core/src/test/scala/org/apache/spark/sql/DatasetBenchmark.scala 85 additions, 27 deletions...rc/test/scala/org/apache/spark/sql/DatasetBenchmark.scala
- sql/core/src/test/scala/org/apache/spark/sql/QueryTest.scala 2 additions, 0 deletionssql/core/src/test/scala/org/apache/spark/sql/QueryTest.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSuite.scala 14 additions, 0 deletions...g/apache/spark/sql/execution/WholeStageCodegenSuite.scala
Loading
Please register or sign in to comment