-
- Downloads
[SPARK-4553] [SPARK-5767] [SQL] Wires Parquet data source with the newly...
[SPARK-4553] [SPARK-5767] [SQL] Wires Parquet data source with the newly introduced write support for data source API This PR migrates the Parquet data source to the new data source write support API. Now users can also overwriting and appending to existing tables. Notice that inserting into partitioned tables is not supported yet. When Parquet data source is enabled, insertion to Hive Metastore Parquet tables is also fullfilled by the Parquet data source. This is done by the newly introduced `HiveMetastoreCatalog.ParquetConversions` rule, which is a "proper" implementation of the original hacky `HiveStrategies.ParquetConversion`. The latter is still preserved, and can be removed together with the old Parquet support in the future. TODO: - [x] Update outdated comments in `newParquet.scala`. <!-- Reviewable:start --> [<img src="https://reviewable.io/review_button.png" height=40 alt="Review on Reviewable"/>](https://reviewable.io/reviews/apache/spark/4563) <!-- Reviewable:end --> Author: Cheng Lian <lian@databricks.com> Closes #4563 from liancheng/parquet-refining and squashes the following commits: fa98d27 [Cheng Lian] Fixes test cases which should disable off Parquet data source 2476e82 [Cheng Lian] Fixes compilation error introduced during rebasing a83d290 [Cheng Lian] Passes Hive Metastore partitioning information to ParquetRelation2
Showing
- sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala 1 addition, 1 deletion...core/src/main/scala/org/apache/spark/sql/SQLContext.scala
- sql/core/src/main/scala/org/apache/spark/sql/parquet/ParquetTableOperations.scala 1 addition, 1 deletion...org/apache/spark/sql/parquet/ParquetTableOperations.scala
- sql/core/src/main/scala/org/apache/spark/sql/parquet/ParquetTest.scala 30 additions, 8 deletions...main/scala/org/apache/spark/sql/parquet/ParquetTest.scala
- sql/core/src/main/scala/org/apache/spark/sql/parquet/newParquet.scala 164 additions, 101 deletions.../main/scala/org/apache/spark/sql/parquet/newParquet.scala
- sql/core/src/test/scala/org/apache/spark/sql/parquet/ParquetFilterSuite.scala 179 additions, 164 deletions...ala/org/apache/spark/sql/parquet/ParquetFilterSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/parquet/ParquetIOSuite.scala 235 additions, 182 deletions...t/scala/org/apache/spark/sql/parquet/ParquetIOSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/parquet/ParquetPartitionDiscoverySuite.scala 199 additions, 3 deletions...he/spark/sql/parquet/ParquetPartitionDiscoverySuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/parquet/ParquetQuerySuite.scala 82 additions, 65 deletions...cala/org/apache/spark/sql/parquet/ParquetQuerySuite.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveContext.scala 5 additions, 6 deletions...rc/main/scala/org/apache/spark/sql/hive/HiveContext.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala 90 additions, 37 deletions...cala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveStrategies.scala 13 additions, 9 deletions...main/scala/org/apache/spark/sql/hive/HiveStrategies.scala
- sql/hive/src/test/scala/org/apache/spark/sql/parquet/parquetSuites.scala 149 additions, 98 deletions...st/scala/org/apache/spark/sql/parquet/parquetSuites.scala
Loading
Please register or sign in to comment