-
- Downloads
[SPARK-19887][SQL] dynamic partition keys can be null or empty string
## What changes were proposed in this pull request? When dynamic partition value is null or empty string, we should write the data to a directory like `a=__HIVE_DEFAULT_PARTITION__`, when we read the data back, we should respect this special directory name and treat it as null. This is the same behavior of impala, see https://issues.apache.org/jira/browse/IMPALA-252 ## How was this patch tested? new regression test Author: Wenchen Fan <wenchen@databricks.com> Closes #17277 from cloud-fan/partition.
Showing
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/ExternalCatalogUtils.scala 1 addition, 1 deletion...che/spark/sql/catalyst/catalog/ExternalCatalogUtils.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala 7 additions, 2 deletions...ala/org/apache/spark/sql/catalyst/catalog/interface.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala 1 addition, 1 deletion...a/org/apache/spark/sql/execution/DataSourceScanExec.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatWriter.scala 4 additions, 7 deletions...he/spark/sql/execution/datasources/FileFormatWriter.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala 1 addition, 2 deletions...e/spark/sql/execution/datasources/PartitioningUtils.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala 2 additions, 2 deletions...scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/PartitionProviderCompatibilitySuite.scala 23 additions, 1 deletion.../spark/sql/hive/PartitionProviderCompatibilitySuite.scala
Loading
Please register or sign in to comment