-
- Downloads
[SPARK-9050] [SQL] Remove unused newOrdering argument from Exchange (cleanup after SPARK-8317)
SPARK-8317 changed the SQL Exchange operator so that it no longer pushed sorting into Spark's shuffle layer, a change which allowed more efficient SQL-specific sorters to be used. This patch performs some leftover cleanup based on those changes: - Exchange's constructor should no longer accept a `newOrdering` since it's no longer used and no longer works as expected. - `addOperatorsIfNecessary` looked at shuffle input's output ordering to decide whether to sort, but this is the wrong node to be examining: it needs to look at whether the post-shuffle node has the right ordering, since shuffling will not preserve row orderings. Thanks to davies for spotting this. Author: Josh Rosen <joshrosen@databricks.com> Closes #7407 from JoshRosen/SPARK-9050 and squashes the following commits: e70be50 [Josh Rosen] No need to wrap line e866494 [Josh Rosen] Refactor addOperatorsIfNecessary to make code clearer 2e467da [Josh Rosen] Remove `newOrdering` from Exchange.
Showing
- sql/core/src/main/scala/org/apache/spark/sql/execution/Exchange.scala 15 additions, 22 deletions.../main/scala/org/apache/spark/sql/execution/Exchange.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala 1 addition, 2 deletions...cala/org/apache/spark/sql/execution/SparkStrategies.scala
Loading
Please register or sign in to comment