-
- Downloads
Spark parquet improvements
A few improvements to the Parquet support for SQL queries: - Instead of files a ParquetRelation is now backed by a directory, which simplifies importing data from other sources - InsertIntoParquetTable operation now supports switching between overwriting or appending (at least in HiveQL) - tests now use the new API - Parquet logging can be set to WARNING level (Default) - Default compression for Parquet files (GZIP, as in parquet-mr) Author: Andre Schumacher <andre.schumacher@iki.fi> Closes #195 from AndreSchumacher/spark_parquet_improvements and squashes the following commits: 54df314 [Andre Schumacher] SPARK-1383 [SQL] Improvements to ParquetRelation
Showing
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/SqlParser.scala 13 additions, 1 deletion.../main/scala/org/apache/spark/sql/catalyst/SqlParser.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Catalog.scala 23 additions, 3 deletions...cala/org/apache/spark/sql/catalyst/analysis/Catalog.scala
- sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala 2 additions, 2 deletions...core/src/main/scala/org/apache/spark/sql/SQLContext.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala 3 additions, 3 deletions...cala/org/apache/spark/sql/execution/SparkStrategies.scala
- sql/core/src/main/scala/org/apache/spark/sql/parquet/ParquetRelation.scala 89 additions, 40 deletions.../scala/org/apache/spark/sql/parquet/ParquetRelation.scala
- sql/core/src/main/scala/org/apache/spark/sql/parquet/ParquetTableOperations.scala 112 additions, 27 deletions...org/apache/spark/sql/parquet/ParquetTableOperations.scala
- sql/core/src/main/scala/org/apache/spark/sql/parquet/ParquetTableSupport.scala 21 additions, 14 deletions...la/org/apache/spark/sql/parquet/ParquetTableSupport.scala
- sql/core/src/main/scala/org/apache/spark/sql/parquet/ParquetTestData.scala 5 additions, 5 deletions.../scala/org/apache/spark/sql/parquet/ParquetTestData.scala
- sql/core/src/test/resources/log4j.properties 3 additions, 5 deletionssql/core/src/test/resources/log4j.properties
- sql/core/src/test/scala/org/apache/spark/sql/parquet/ParquetQuerySuite.scala 100 additions, 18 deletions...cala/org/apache/spark/sql/parquet/ParquetQuerySuite.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala 2 additions, 0 deletions...cala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/TestHive.scala 2 additions, 0 deletions...e/src/main/scala/org/apache/spark/sql/hive/TestHive.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/CachedTableSuite.scala 2 additions, 2 deletions...st/scala/org/apache/spark/sql/hive/CachedTableSuite.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveComparisonTest.scala 3 additions, 3 deletions.../apache/spark/sql/hive/execution/HiveComparisonTest.scala
- sql/hive/src/test/scala/org/apache/spark/sql/parquet/HiveParquetSuite.scala 80 additions, 89 deletions...scala/org/apache/spark/sql/parquet/HiveParquetSuite.scala
Loading
Please register or sign in to comment