-
- Downloads
[SPARK-17990][SPARK-18302][SQL] correct several partition related behaviours of ExternalCatalog
## What changes were proposed in this pull request? This PR corrects several partition related behaviors of `ExternalCatalog`: 1. default partition location should not always lower case the partition column names in path string(fix `HiveExternalCatalog`) 2. rename partition should not always lower case the partition column names in updated partition path string(fix `HiveExternalCatalog`) 3. rename partition should update the partition location only for managed table(fix `InMemoryCatalog`) 4. create partition with existing directory should be fine(fix `InMemoryCatalog`) 5. create partition with non-existing directory should create that directory(fix `InMemoryCatalog`) 6. drop partition from external table should not delete the directory(fix `InMemoryCatalog`) ## How was this patch tested? new tests in `ExternalCatalogSuite` Author: Wenchen Fan <wenchen@databricks.com> Closes #15797 from cloud-fan/partition.
Showing
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/ExternalCatalogUtils.scala 121 additions, 0 deletions...che/spark/sql/catalyst/catalog/ExternalCatalogUtils.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/InMemoryCatalog.scala 46 additions, 46 deletions...g/apache/spark/sql/catalyst/catalog/InMemoryCatalog.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala 11 additions, 0 deletions...ala/org/apache/spark/sql/catalyst/catalog/interface.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/catalog/ExternalCatalogSuite.scala 119 additions, 31 deletions...che/spark/sql/catalyst/catalog/ExternalCatalogSuite.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalogSuite.scala 17 additions, 7 deletions...ache/spark/sql/catalyst/catalog/SessionCatalogSuite.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala 4 additions, 4 deletions...in/scala/org/apache/spark/sql/execution/command/ddl.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala 2 additions, 1 deletion...scala/org/apache/spark/sql/execution/command/tables.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/CatalogFileIndex.scala 1 addition, 1 deletion...he/spark/sql/execution/datasources/CatalogFileIndex.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala 1 addition, 1 deletion.../spark/sql/execution/datasources/DataSourceStrategy.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatWriter.scala 3 additions, 3 deletions...he/spark/sql/execution/datasources/FileFormatWriter.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningAwareFileIndex.scala 0 additions, 2 deletions...ql/execution/datasources/PartitioningAwareFileIndex.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala 8 additions, 86 deletions...e/spark/sql/execution/datasources/PartitioningUtils.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala 4 additions, 4 deletions...ala/org/apache/spark/sql/execution/command/DDLSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetPartitionDiscoverySuite.scala 7 additions, 14 deletions.../datasources/parquet/ParquetPartitionDiscoverySuite.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala 48 additions, 3 deletions...scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveSparkSubmitSuite.scala 2 additions, 2 deletions...cala/org/apache/spark/sql/hive/HiveSparkSubmitSuite.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/MultiDatabaseSuite.scala 1 addition, 1 deletion.../scala/org/apache/spark/sql/hive/MultiDatabaseSuite.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala 1 addition, 1 deletion...la/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala 1 addition, 1 deletion...a/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala
Loading
Please register or sign in to comment