-
- Downloads
[SPARK-5852][SQL]Fail to convert a newly created empty metastore parquet table...
[SPARK-5852][SQL]Fail to convert a newly created empty metastore parquet table to a data source parquet table. The problem is that after we create an empty hive metastore parquet table (e.g. `CREATE TABLE test (a int) STORED AS PARQUET`), Hive will create an empty dir for us, which cause our data source `ParquetRelation2` fail to get the schema of the table. See JIRA for the case to reproduce the bug and the exception. This PR is based on #4562 from chenghao-intel. JIRA: https://issues.apache.org/jira/browse/SPARK-5852 Author: Yin Huai <yhuai@databricks.com> Author: Cheng Hao <hao.cheng@intel.com> Closes #4655 from yhuai/CTASParquet and squashes the following commits: b8b3450 [Yin Huai] Update tests. 2ac94f7 [Yin Huai] Update tests. 3db3d20 [Yin Huai] Minor update. d7e2308 [Yin Huai] Revert changes in HiveMetastoreCatalog.scala. 36978d1 [Cheng Hao] Update the code as feedback a04930b [Cheng Hao] fix bug of scan an empty parquet based table 442ffe0 [Cheng Hao] passdown the schema for Parquet File in HiveContext
Showing
- sql/core/src/main/scala/org/apache/spark/sql/parquet/newParquet.scala 14 additions, 4 deletions.../main/scala/org/apache/spark/sql/parquet/newParquet.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/MetastoreDataSourcesSuite.scala 38 additions, 0 deletions...org/apache/spark/sql/hive/MetastoreDataSourcesSuite.scala
- sql/hive/src/test/scala/org/apache/spark/sql/parquet/parquetSuites.scala 112 additions, 2 deletions...st/scala/org/apache/spark/sql/parquet/parquetSuites.scala
Loading
Please register or sign in to comment