-
- Downloads
[SPARK-19887][SQL] dynamic partition keys can be null or empty string
When dynamic partition value is null or empty string, we should write the data to a directory like `a=__HIVE_DEFAULT_PARTITION__`, when we read the data back, we should respect this special directory name and treat it as null. This is the same behavior of impala, see https://issues.apache.org/jira/browse/IMPALA-252 new regression test Author: Wenchen Fan <wenchen@databricks.com> Closes #17277 from cloud-fan/partition. (cherry picked from commit dacc382f) Signed-off-by:Wenchen Fan <wenchen@databricks.com>
Showing
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/ExternalCatalogUtils.scala 10 additions, 7 deletions...che/spark/sql/catalyst/catalog/ExternalCatalogUtils.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala 7 additions, 2 deletions...ala/org/apache/spark/sql/catalyst/catalog/interface.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala 1 addition, 1 deletion...a/org/apache/spark/sql/execution/DataSourceScanExec.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatWriter.scala 4 additions, 7 deletions...he/spark/sql/execution/datasources/FileFormatWriter.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala 1 addition, 1 deletion...e/spark/sql/execution/datasources/PartitioningUtils.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala 2 additions, 2 deletions...scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/PartitionProviderCompatibilitySuite.scala 24 additions, 1 deletion.../spark/sql/hive/PartitionProviderCompatibilitySuite.scala
Loading
Please register or sign in to comment