-
- Downloads
[SPARK-5501][SPARK-5420][SQL] Write support for the data source API
This PR aims to support `INSERT INTO/OVERWRITE TABLE tableName` and `CREATE TABLE tableName AS SELECT` for the data source API (partitioned tables are not supported). In this PR, I am also adding the support of `IF NOT EXISTS` for our ddl parser. The current semantic of `IF NOT EXISTS` is explained as follows. * For a `CREATE TEMPORARY TABLE` statement, it does not `IF NOT EXISTS` for now. * For a `CREATE TABLE` statement (we are creating a metastore table), if there is an existing table having the same name ... * when `IF NOT EXISTS` clause is used, we will do nothing. * when `IF NOT EXISTS` clause is not used, the user will see an exception saying the table already exists. TODOs: - [x] CTAS support - [x] Programmatic APIs - [ ] Python API (another PR) - [x] More unit tests - [ ] Documents (another PR) marmbrus liancheng rxin Author: Yin Huai <yhuai@databricks.com> Closes #4294 from yhuai/writeSupport and squashes the following commits: 3db1539 [Yin Huai] save does not take overwrite. 1c98881 [Yin Huai] Fix test. 142372a [Yin Huai] Merge remote-tracking branch 'upstream/master' into writeSupport 34e1bfb [Yin Huai] Address comments. 1682ca6 [Yin Huai] Better support for CTAS statements. e789d64 [Yin Huai] For the Scala API, let users to use tuples to provide options. 0128065 [Yin Huai] Short hand versions of save and load. 66ebd74 [Yin Huai] Formatting. 9203ec2 [Yin Huai] Merge remote-tracking branch 'upstream/master' into writeSupport e5d29f2 [Yin Huai] Programmatic APIs. 1a719a5 [Yin Huai] CREATE TEMPORARY TABLE with IF NOT EXISTS is not allowed for now. 909924f [Yin Huai] Add saveAsTable for the data source API to DataFrame. 95a7c71 [Yin Huai] Fix bug when handling IF NOT EXISTS clause in a CREATE TEMPORARY TABLE statement. d37b19c [Yin Huai] Cheng's comments. fd6758c [Yin Huai] Use BeforeAndAfterAll. 7880891 [Yin Huai] Support CREATE TABLE AS SELECT STATEMENT and the IF NOT EXISTS clause. cb85b05 [Yin Huai] Initial write support. 2f91354 [Yin Huai] Make INSERT OVERWRITE/INTO statements consistent between HiveQL and SqlParser.
Showing
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/SqlParser.scala 2 additions, 2 deletions.../main/scala/org/apache/spark/sql/catalyst/SqlParser.scala
- sql/core/src/main/scala/org/apache/spark/sql/DataFrame.scala 47 additions, 2 deletionssql/core/src/main/scala/org/apache/spark/sql/DataFrame.scala
- sql/core/src/main/scala/org/apache/spark/sql/DataFrameImpl.scala 56 additions, 2 deletions...e/src/main/scala/org/apache/spark/sql/DataFrameImpl.scala
- sql/core/src/main/scala/org/apache/spark/sql/IncomputableColumn.scala 22 additions, 0 deletions.../main/scala/org/apache/spark/sql/IncomputableColumn.scala
- sql/core/src/main/scala/org/apache/spark/sql/SQLConf.scala 6 additions, 0 deletionssql/core/src/main/scala/org/apache/spark/sql/SQLConf.scala
- sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala 25 additions, 1 deletion...core/src/main/scala/org/apache/spark/sql/SQLContext.scala
- sql/core/src/main/scala/org/apache/spark/sql/api.scala 27 additions, 0 deletionssql/core/src/main/scala/org/apache/spark/sql/api.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala 26 additions, 5 deletions...cala/org/apache/spark/sql/execution/SparkStrategies.scala
- sql/core/src/main/scala/org/apache/spark/sql/json/JSONRelation.scala 50 additions, 9 deletions...c/main/scala/org/apache/spark/sql/json/JSONRelation.scala
- sql/core/src/main/scala/org/apache/spark/sql/sources/DataSourceStrategy.scala 8 additions, 1 deletion...ala/org/apache/spark/sql/sources/DataSourceStrategy.scala
- sql/core/src/main/scala/org/apache/spark/sql/sources/commands.scala 35 additions, 0 deletions...rc/main/scala/org/apache/spark/sql/sources/commands.scala
- sql/core/src/main/scala/org/apache/spark/sql/sources/ddl.scala 112 additions, 14 deletions...ore/src/main/scala/org/apache/spark/sql/sources/ddl.scala
- sql/core/src/main/scala/org/apache/spark/sql/sources/interfaces.scala 18 additions, 5 deletions.../main/scala/org/apache/spark/sql/sources/interfaces.scala
- sql/core/src/test/scala/org/apache/spark/sql/parquet/ParquetQuerySuite.scala 11 additions, 1 deletion...cala/org/apache/spark/sql/parquet/ParquetQuerySuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/sources/CreateTableAsSelectSuite.scala 147 additions, 0 deletions...g/apache/spark/sql/sources/CreateTableAsSelectSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/sources/InsertIntoSuite.scala 96 additions, 0 deletions.../scala/org/apache/spark/sql/sources/InsertIntoSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/sources/SaveLoadSuite.scala 88 additions, 0 deletions...st/scala/org/apache/spark/sql/sources/SaveLoadSuite.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveContext.scala 66 additions, 1 deletion...rc/main/scala/org/apache/spark/sql/hive/HiveContext.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala 28 additions, 7 deletions...cala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveStrategies.scala 15 additions, 3 deletions...main/scala/org/apache/spark/sql/hive/HiveStrategies.scala
Loading
Please register or sign in to comment