-
- Downloads
[SPARK-13031][SQL] cleanup codegen and improve test coverage
1. enable whole stage codegen during tests even there is only one operator supports that. 2. split doProduce() into two APIs: upstream() and doProduce() 3. generate prefix for fresh names of each operator 4. pass UnsafeRow to parent directly (avoid getters and create UnsafeRow again) 5. fix bugs and tests. This PR re-open #10944 and fix the bug. Author: Davies Liu <davies@databricks.com> Closes #10977 from davies/gen_refactor.
Showing
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala 11 additions, 2 deletions...park/sql/catalyst/expressions/codegen/CodeGenerator.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateMutableProjection.scala 1 addition, 1 deletion...alyst/expressions/codegen/GenerateMutableProjection.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/WholeStageCodegen.scala 131 additions, 57 deletions...la/org/apache/spark/sql/execution/WholeStageCodegen.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/AggregationIterator.scala 1 addition, 1 deletion...e/spark/sql/execution/aggregate/AggregationIterator.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/TungstenAggregate.scala 68 additions, 30 deletions...che/spark/sql/execution/aggregate/TungstenAggregate.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/basicOperators.scala 53 additions, 43 deletions...scala/org/apache/spark/sql/execution/basicOperators.scala
- sql/core/src/test/scala/org/apache/spark/sql/DataFrameAggregateSuite.scala 7 additions, 0 deletions.../scala/org/apache/spark/sql/DataFrameAggregateSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala 53 additions, 50 deletions...e/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/metric/SQLMetricsSuite.scala 18 additions, 16 deletions...g/apache/spark/sql/execution/metric/SQLMetricsSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/test/SQLTestUtils.scala 1 addition, 1 deletion...c/test/scala/org/apache/spark/sql/test/SQLTestUtils.scala
- sql/core/src/test/scala/org/apache/spark/sql/util/DataFrameCallbackSuite.scala 6 additions, 4 deletions...la/org/apache/spark/sql/util/DataFrameCallbackSuite.scala
Loading
Please register or sign in to comment