-
- Downloads
[SPARK-18038][SQL] Move output partitioning definition from UnaryNodeExec to its children
## What changes were proposed in this pull request? Jira : https://issues.apache.org/jira/browse/SPARK-18038 This was a suggestion by rxin over one of the dev list discussion : http://apache-spark-developers-list.1001551.n3.nabble.com/Project-not-preserving-child-partitioning-td19417.html His words: >> It would be better (safer) to move the output partitioning definition into each of the operator and remove it from UnaryExecNode. With this PR, following is the output partitioning and ordering for all the impls of `UnaryExecNode`. UnaryExecNode's impl | outputPartitioning | outputOrdering | comment ------------ | ------------- | ------------ | ------------ AppendColumnsExec | child's | Nil | child's ordering can be used AppendColumnsWithObjectExec | child's | Nil | child's ordering can be used BroadcastExchangeExec | BroadcastPartitioning | Nil | - CoalesceExec | UnknownPartitioning | Nil | - CollectLimitExec | SinglePartition | Nil | - DebugExec | child's | Nil | child's ordering can be used DeserializeToObjectExec | child's | Nil | child's ordering can be used ExpandExec | UnknownPartitioning | Nil | - FilterExec | child's | child's | - FlatMapGroupsInRExec | child's | Nil | child's ordering can be used GenerateExec | child's | Nil | need to dig more GlobalLimitExec | child's | child's | - HashAggregateExec | child's | Nil | - InputAdapter | child's | child's | - InsertIntoHiveTable | child's | Nil | terminal node, doesn't need partitioning LocalLimitExec | child's | child's | - MapElementsExec | child's | child's | - MapGroupsExec | child's | Nil | child's ordering can be used MapPartitionsExec | child's | Nil | child's ordering can be used ProjectExec | child's | child's | - SampleExec | child's | Nil | child's ordering can be used ScriptTransformation | child's | Nil | child's ordering can be used SerializeFromObjectExec | child's | Nil | child's ordering can be used ShuffleExchange | custom | Nil | - SortAggregateExec | child's | sort over grouped exprs | - SortExec | child's | custom | - StateStoreRestoreExec | child's | Nil | child's ordering can be used StateStoreSaveExec | child's | Nil | child's ordering can be used SubqueryExec | child's | child's | - TakeOrderedAndProjectExec | SinglePartition | custom | - WholeStageCodegenExec | child's | child's | - WindowExec | child's | child's | - ## How was this patch tested? This does NOT change any existing functionality so relying on existing tests Author: Tejas Patil <tejasp@fb.com> Closes #15575 from tejasapatil/SPARK-18038_UnaryNodeExec_output_partitioning.
Showing
- sql/core/src/main/scala/org/apache/spark/sql/execution/GenerateExec.scala 3 additions, 0 deletions...n/scala/org/apache/spark/sql/execution/GenerateExec.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/SortExec.scala 5 additions, 1 deletion.../main/scala/org/apache/spark/sql/execution/SortExec.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/SparkPlan.scala 0 additions, 2 deletions...main/scala/org/apache/spark/sql/execution/SparkPlan.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/WholeStageCodegenExec.scala 4 additions, 0 deletions...rg/apache/spark/sql/execution/WholeStageCodegenExec.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/HashAggregateExec.scala 2 additions, 0 deletions...che/spark/sql/execution/aggregate/HashAggregateExec.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/SortAggregateExec.scala 3 additions, 1 deletion...che/spark/sql/execution/aggregate/SortAggregateExec.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/basicPhysicalOperators.scala 8 additions, 0 deletions...g/apache/spark/sql/execution/basicPhysicalOperators.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/debug/package.scala 3 additions, 1 deletion.../scala/org/apache/spark/sql/execution/debug/package.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/limit.scala 11 additions, 5 deletions...src/main/scala/org/apache/spark/sql/execution/limit.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/objects.scala 17 additions, 0 deletions...c/main/scala/org/apache/spark/sql/execution/objects.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StatefulAggregate.scala 6 additions, 0 deletions...che/spark/sql/execution/streaming/StatefulAggregate.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/window/WindowExec.scala 2 additions, 0 deletions...la/org/apache/spark/sql/execution/window/WindowExec.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/ReferenceSort.scala 2 additions, 0 deletions.../scala/org/apache/spark/sql/execution/ReferenceSort.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala 3 additions, 1 deletion...apache/spark/sql/hive/execution/InsertIntoHiveTable.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/ScriptTransformation.scala 3 additions, 0 deletions...pache/spark/sql/hive/execution/ScriptTransformation.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/ScriptTransformationSuite.scala 4 additions, 0 deletions.../spark/sql/hive/execution/ScriptTransformationSuite.scala
Loading
Please register or sign in to comment