-
- Downloads
[SPARK-16119][SQL] Support PURGE option to drop table / partition.
This option is used by Hive to directly delete the files instead of moving them to the trash. This is needed in certain configurations where moving the files does not work. For non-Hive tables and partitions, Spark already behaves as if the PURGE option was set, so there's no need to do anything. Hive support for PURGE was added in 0.14 (for tables) and 1.2 (for partitions), so the code reflects that: trying to use the option with older versions of Hive will cause an exception to be thrown. The change is a little noisier than I would like, because of the code to propagate the new flag through all the interfaces and implementations; the main changes are in the parser and in HiveShim, aside from the tests (DDLCommandSuite, VersionsSuite). Tested by running sql and catalyst unit tests, plus VersionsSuite which has been updated to test the version-specific behavior. I also ran an internal test suite that uses PURGE and would not pass previously. Author: Marcelo Vanzin <vanzin@cloudera.com> Closes #13831 from vanzin/SPARK-16119.
Showing
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/ExternalCatalog.scala 3 additions, 2 deletions...g/apache/spark/sql/catalyst/catalog/ExternalCatalog.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/InMemoryCatalog.scala 4 additions, 2 deletions...g/apache/spark/sql/catalyst/catalog/InMemoryCatalog.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala 9 additions, 5 deletions...rg/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/catalog/ExternalCatalogSuite.scala 16 additions, 15 deletions...che/spark/sql/catalyst/catalog/ExternalCatalogSuite.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalogSuite.scala 36 additions, 21 deletions...ache/spark/sql/catalyst/catalog/SessionCatalogSuite.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala 4 additions, 8 deletions...scala/org/apache/spark/sql/execution/SparkSqlParser.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala 6 additions, 4 deletions...in/scala/org/apache/spark/sql/execution/command/ddl.scala
- sql/core/src/main/scala/org/apache/spark/sql/internal/CatalogImpl.scala 1 addition, 1 deletion...ain/scala/org/apache/spark/sql/internal/CatalogImpl.scala
- sql/core/src/test/scala/org/apache/spark/sql/SQLContextSuite.scala 2 additions, 2 deletions...src/test/scala/org/apache/spark/sql/SQLContextSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLCommandSuite.scala 37 additions, 26 deletions.../apache/spark/sql/execution/command/DDLCommandSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala 1 addition, 1 deletion...ala/org/apache/spark/sql/execution/command/DDLSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetQuerySuite.scala 2 additions, 2 deletions...sql/execution/datasources/parquet/ParquetQuerySuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/internal/CatalogSuite.scala 1 addition, 1 deletion...st/scala/org/apache/spark/sql/internal/CatalogSuite.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala 6 additions, 4 deletions...scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClient.scala 3 additions, 2 deletions...n/scala/org/apache/spark/sql/hive/client/HiveClient.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala 6 additions, 4 deletions...ala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala 98 additions, 2 deletions...ain/scala/org/apache/spark/sql/hive/client/HiveShim.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/CreateHiveTableAsSelectCommand.scala 2 additions, 1 deletion...k/sql/hive/execution/CreateHiveTableAsSelectCommand.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/ListTablesSuite.scala 1 addition, 1 deletion...est/scala/org/apache/spark/sql/hive/ListTablesSuite.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/StatisticsSuite.scala 1 addition, 1 deletion...est/scala/org/apache/spark/sql/hive/StatisticsSuite.scala
Loading
Please register or sign in to comment