-
- Downloads
[SPARK-15646][SQL] When spark.sql.hive.convertCTAS is true, the conversion...
[SPARK-15646][SQL] When spark.sql.hive.convertCTAS is true, the conversion rule needs to respect TEXTFILE/SEQUENCEFILE format and the user-defined location ## What changes were proposed in this pull request? When `spark.sql.hive.convertCTAS` is true, for a CTAS statement, we will create a data source table using the default source (i.e. parquet) if the CTAS does not specify any Hive storage format. However, there are two issues with this conversion logic. 1. First, we determine if a CTAS statement defines storage format by checking the serde. However, TEXTFILE/SEQUENCEFILE does not have a default serde. When we do the check, we have not set the default serde. So, a query like `CREATE TABLE abc STORED AS TEXTFILE AS SELECT ...` actually creates a data source parquet table. 2. In the conversion logic, we are ignoring the user-specified location. This PR fixes the above two issues. Also, this PR makes the parser throws an exception when a CTAS statement has a PARTITIONED BY clause. This change is made because Hive's syntax does not allow it and our current implementation actually does not work for this case (the insert operation always throws an exception because the insertion does not pick up the partitioning info). ## How was this patch tested? I am adding new tests in SQLQuerySuite and HiveDDLCommandSuite. Author: Yin Huai <yhuai@databricks.com> Closes #13386 from yhuai/SPARK-14507.
Showing
- sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala 35 additions, 2 deletions...scala/org/apache/spark/sql/execution/SparkSqlParser.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala 1 addition, 1 deletion...scala/org/apache/spark/sql/execution/command/tables.scala
- sql/core/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala 10 additions, 0 deletions...rc/main/scala/org/apache/spark/sql/internal/SQLConf.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala 13 additions, 44 deletions...cala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveSessionState.scala 0 additions, 16 deletions...in/scala/org/apache/spark/sql/hive/HiveSessionState.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala 0 additions, 6 deletions.../src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/CreateHiveTableAsSelectCommand.scala 5 additions, 4 deletions...k/sql/hive/execution/CreateHiveTableAsSelectCommand.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveDDLCommandSuite.scala 10 additions, 15 deletions...scala/org/apache/spark/sql/hive/HiveDDLCommandSuite.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveExplainSuite.scala 3 additions, 3 deletions...rg/apache/spark/sql/hive/execution/HiveExplainSuite.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala 99 additions, 36 deletions...a/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala
Loading
Please register or sign in to comment