-
- Downloads
[SPARK-19080][SQL] simplify data source analysis
## What changes were proposed in this pull request? The current way of resolving `InsertIntoTable` and `CreateTable` is convoluted: sometimes we replace them with concrete implementation commands during analysis, sometimes during planning phase. And the error checking logic is also a mess: we may put it in extended analyzer rules, or extended checking rules, or `CheckAnalysis`. This PR simplifies the data source analysis: 1. `InsertIntoTable` and `CreateTable` are always unresolved and need to be replaced by concrete implementation commands during analysis. 2. The error checking logic is mainly in 2 rules: `PreprocessTableCreation` and `PreprocessTableInsertion`. ## How was this patch tested? existing test. Author: Wenchen Fan <wenchen@databricks.com> Closes #16269 from cloud-fan/ddl.
Showing
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala 4 additions, 22 deletions...rg/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/UnsupportedOperationChecker.scala 0 additions, 3 deletions...k/sql/catalyst/analysis/UnsupportedOperationChecker.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala 8 additions, 8 deletions...rk/sql/catalyst/plans/logical/basicLogicalOperators.scala
- sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala 1 addition, 1 deletion...src/main/scala/org/apache/spark/sql/DataFrameWriter.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/SparkPlanner.scala 0 additions, 1 deletion...n/scala/org/apache/spark/sql/execution/SparkPlanner.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala 0 additions, 28 deletions...cala/org/apache/spark/sql/execution/SparkStrategies.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala 4 additions, 2 deletions...scala/org/apache/spark/sql/execution/command/tables.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala 13 additions, 5 deletions.../spark/sql/execution/datasources/DataSourceStrategy.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/ddl.scala 14 additions, 4 deletions...cala/org/apache/spark/sql/execution/datasources/ddl.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/rules.scala 47 additions, 83 deletions...la/org/apache/spark/sql/execution/datasources/rules.scala
- sql/core/src/main/scala/org/apache/spark/sql/internal/SessionState.scala 2 additions, 3 deletions...in/scala/org/apache/spark/sql/internal/SessionState.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala 2 additions, 2 deletions...ala/org/apache/spark/sql/execution/command/DDLSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/sources/InsertSuite.scala 1 addition, 1 deletion...test/scala/org/apache/spark/sql/sources/InsertSuite.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala 8 additions, 16 deletions...cala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveSessionCatalog.scala 2 additions, 0 deletions.../scala/org/apache/spark/sql/hive/HiveSessionCatalog.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveSessionState.scala 4 additions, 5 deletions...in/scala/org/apache/spark/sql/hive/HiveSessionState.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveStrategies.scala 7 additions, 3 deletions...main/scala/org/apache/spark/sql/hive/HiveStrategies.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveDDLCommandSuite.scala 1 addition, 1 deletion...scala/org/apache/spark/sql/hive/HiveDDLCommandSuite.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/InsertIntoHiveTableSuite.scala 2 additions, 1 deletion.../org/apache/spark/sql/hive/InsertIntoHiveTableSuite.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/parquetSuites.scala 6 additions, 10 deletions.../test/scala/org/apache/spark/sql/hive/parquetSuites.scala
Loading
Please register or sign in to comment