-
- Downloads
[SPARK-13254][SQL] Fix planning of TakeOrderedAndProject operator
The patch for SPARK-8964 ("use Exchange to perform shuffle in Limit" / #7334) inadvertently broke the planning of the TakeOrderedAndProject operator: because ReturnAnswer was the new root of the query plan, the TakeOrderedAndProject rule was unable to match before BasicOperators.
This patch fixes this by moving the `TakeOrderedAndCollect` and `CollectLimit` rules into the same strategy.
In addition, I made changes to the TakeOrderedAndProject operator in order to make its `doExecute()` method lazy and added a new TakeOrderedAndProjectSuite which tests the new code path.
/cc davies and marmbrus for review.
Author: Josh Rosen <joshrosen@databricks.com>
Closes #11145 from JoshRosen/take-ordered-and-project-fix.
Showing
- sql/core/src/main/scala/org/apache/spark/sql/execution/SparkPlanner.scala 1 addition, 1 deletion...n/scala/org/apache/spark/sql/execution/SparkPlanner.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala 25 additions, 15 deletions...cala/org/apache/spark/sql/execution/SparkStrategies.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/limit.scala 21 additions, 9 deletions...src/main/scala/org/apache/spark/sql/execution/limit.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/PlannerSuite.scala 26 additions, 18 deletions...t/scala/org/apache/spark/sql/execution/PlannerSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/TakeOrderedAndProjectSuite.scala 85 additions, 0 deletions...ache/spark/sql/execution/TakeOrderedAndProjectSuite.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveContext.scala 1 addition, 1 deletion...rc/main/scala/org/apache/spark/sql/hive/HiveContext.scala
Loading
Please register or sign in to comment