-
- Downloads
[SPARK-19658][SQL] Set NumPartitions of RepartitionByExpression In Parser
### What changes were proposed in this pull request? Currently, if `NumPartitions` is not set in RepartitionByExpression, we will set it using `spark.sql.shuffle.partitions` during Planner. However, this is not following the general resolution process. This PR is to set it in `Parser` and then `Optimizer` can use the value for plan optimization. ### How was this patch tested? Added a test case. Author: Xiao Li <gatorsmile@gmail.com> Closes #16988 from gatorsmile/resolveRepartition.
Showing
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/dsl/package.scala 2 additions, 2 deletions...ain/scala/org/apache/spark/sql/catalyst/dsl/package.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala 1 addition, 1 deletion...a/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala 13 additions, 3 deletions...ala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala 2 additions, 7 deletions...rk/sql/catalyst/plans/logical/basicLogicalOperators.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnalysisSuite.scala 6 additions, 4 deletions...rg/apache/spark/sql/catalyst/analysis/AnalysisSuite.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/PlanParserSuite.scala 1 addition, 4 deletions...rg/apache/spark/sql/catalyst/parser/PlanParserSuite.scala
- sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala 3 additions, 2 deletionssql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala 14 additions, 3 deletions...scala/org/apache/spark/sql/execution/SparkSqlParser.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala 2 additions, 4 deletions...cala/org/apache/spark/sql/execution/SparkStrategies.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/SparkSqlParserSuite.scala 30 additions, 2 deletions.../org/apache/spark/sql/execution/SparkSqlParserSuite.scala
Loading
Please register or sign in to comment