-
- Downloads
[SPARK-19587][SQL] bucket sorting columns should not be picked from partition columns
## What changes were proposed in this pull request? We will throw an exception if bucket columns are part of partition columns, this should also apply to sort columns. This PR also move the checking logic from `DataFrameWriter` to `PreprocessTableCreation`, which is the central place for checking and normailization. ## How was this patch tested? updated test. Author: Wenchen Fan <wenchen@databricks.com> Closes #16931 from cloud-fan/bucket.
Showing
- sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala 2 additions, 38 deletions...src/main/scala/org/apache/spark/sql/DataFrameWriter.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/rules.scala 15 additions, 3 deletions...la/org/apache/spark/sql/execution/datasources/rules.scala
- sql/hive/src/test/scala/org/apache/spark/sql/sources/BucketedWriteSuite.scala 8 additions, 7 deletions...ala/org/apache/spark/sql/sources/BucketedWriteSuite.scala
Loading
Please register or sign in to comment