-
- Downloads
[SPARK-14491] [SQL] refactor object operator framework to make it easy to eliminate serializations
## What changes were proposed in this pull request? This PR tries to separate the serialization and deserialization logic from object operators, so that it's easier to eliminate unnecessary serializations in optimizer. Typed aggregate related operators are special, they will deserialize the input row to multiple objects and it's difficult to simply use a deserializer operator to abstract it, so we still mix the deserialization logic there. ## How was this patch tested? existing tests and new test in `EliminateSerializationSuite` Author: Wenchen Fan <wenchen@databricks.com> Closes #12260 from cloud-fan/encoder.
Showing
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala 3 additions, 3 deletions...ala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/dsl/package.scala 4 additions, 0 deletions...ain/scala/org/apache/spark/sql/catalyst/dsl/package.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala 9 additions, 22 deletions...a/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/object.scala 82 additions, 84 deletions.../org/apache/spark/sql/catalyst/plans/logical/object.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/EliminateSerializationSuite.scala 33 additions, 29 deletions.../sql/catalyst/optimizer/EliminateSerializationSuite.scala
- sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala 5 additions, 5 deletionssql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala 12 additions, 10 deletions...cala/org/apache/spark/sql/execution/SparkStrategies.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/WholeStageCodegen.scala 4 additions, 0 deletions...la/org/apache/spark/sql/execution/WholeStageCodegen.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/objects.scala 98 additions, 62 deletions...c/main/scala/org/apache/spark/sql/execution/objects.scala
- sql/core/src/test/scala/org/apache/spark/sql/QueryTest.scala 3 additions, 1 deletionsql/core/src/test/scala/org/apache/spark/sql/QueryTest.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSuite.scala 1 addition, 1 deletion...g/apache/spark/sql/execution/WholeStageCodegenSuite.scala
Loading
Please register or sign in to comment