-
- Downloads
[SPARK-18631][SQL] Changed ExchangeCoordinator re-partitioning to avoid more data skew
## What changes were proposed in this pull request? Re-partitioning logic in ExchangeCoordinator changed so that adding another pre-shuffle partition to the post-shuffle partition will not be done if doing so would cause the size of the post-shuffle partition to exceed the target partition size. ## How was this patch tested? Existing tests updated to reflect new expectations. Author: Mark Hamstra <markhamstra@gmail.com> Closes #16065 from markhamstra/SPARK-17064.
Showing
- sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/ExchangeCoordinator.scala 15 additions, 17 deletions...he/spark/sql/execution/exchange/ExchangeCoordinator.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/ExchangeCoordinatorSuite.scala 20 additions, 20 deletions...apache/spark/sql/execution/ExchangeCoordinatorSuite.scala
Loading
Please register or sign in to comment