-
- Downloads
[SPARK-17970][SQL] store partition spec in metastore for data source table
## What changes were proposed in this pull request? We should follow hive table and also store partition spec in metastore for data source table. This brings 2 benefits: 1. It's more flexible to manage the table data files, as users can use `ADD PARTITION`, `DROP PARTITION` and `RENAME PARTITION` 2. We don't need to cache all file status for data source table anymore. ## How was this patch tested? existing tests. Author: Eric Liang <ekl@databricks.com> Author: Michael Allman <michael@videoamp.com> Author: Eric Liang <ekhliang@gmail.com> Author: Wenchen Fan <wenchen@databricks.com> Closes #15515 from cloud-fan/partition.
Showing
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala 8 additions, 4 deletions...ala/org/apache/spark/sql/catalyst/catalog/interface.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/trees/TreeNodeSuite.scala 1 addition, 0 deletions...a/org/apache/spark/sql/catalyst/trees/TreeNodeSuite.scala
- sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala 11 additions, 2 deletions...src/main/scala/org/apache/spark/sql/DataFrameWriter.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/command/AnalyzeColumnCommand.scala 2 additions, 1 deletion...he/spark/sql/execution/command/AnalyzeColumnCommand.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/command/AnalyzeTableCommand.scala 2 additions, 1 deletion...che/spark/sql/execution/command/AnalyzeTableCommand.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/command/createDataSourceTables.scala 16 additions, 1 deletion.../spark/sql/execution/command/createDataSourceTables.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala 54 additions, 36 deletions...in/scala/org/apache/spark/sql/execution/command/ddl.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala 14 additions, 25 deletions...scala/org/apache/spark/sql/execution/command/tables.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala 15 additions, 5 deletions...g/apache/spark/sql/execution/datasources/DataSource.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala 12 additions, 3 deletions.../spark/sql/execution/datasources/DataSourceStrategy.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileCatalog.scala 4 additions, 0 deletions.../apache/spark/sql/execution/datasources/FileCatalog.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileStatusCache.scala 1 addition, 1 deletion...che/spark/sql/execution/datasources/FileStatusCache.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningAwareFileCatalog.scala 7 additions, 5 deletions.../execution/datasources/PartitioningAwareFileCatalog.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/TableFileCatalog.scala 3 additions, 1 deletion...he/spark/sql/execution/datasources/TableFileCatalog.scala
- sql/core/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala 9 additions, 7 deletions...rc/main/scala/org/apache/spark/sql/internal/SQLConf.scala
- sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala 1 addition, 1 deletion...c/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala 78 additions, 122 deletions...ala/org/apache/spark/sql/execution/command/DDLSuite.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala 85 additions, 44 deletions...scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala 4 additions, 5 deletions...cala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala 1 addition, 4 deletions...ala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
Loading
Please register or sign in to comment