-
- Downloads
[SPARK-13543][SQL] Support for specifying compression codec for Parquet/ORC via option()
## What changes were proposed in this pull request? This PR adds the support to specify compression codecs for both ORC and Parquet. ## How was this patch tested? unittests within IDE and code style tests with `dev/run_tests`. Author: hyukjinkwon <gurwls223@gmail.com> Closes #11464 from HyukjinKwon/SPARK-13543.
Showing
- python/pyspark/sql/readwriter.py 34 additions, 21 deletionspython/pyspark/sql/readwriter.py
- sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala 16 additions, 3 deletions...src/main/scala/org/apache/spark/sql/DataFrameWriter.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/CompressionCodecs.scala 16 additions, 6 deletions...e/spark/sql/execution/datasources/CompressionCodecs.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetRelation.scala 15 additions, 1 deletion...k/sql/execution/datasources/parquet/ParquetRelation.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala 46 additions, 0 deletions...apache/spark/sql/execution/datasources/csv/CSVSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala 45 additions, 1 deletion...ache/spark/sql/execution/datasources/json/JsonSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/text/TextSuite.scala 46 additions, 9 deletions...ache/spark/sql/execution/datasources/text/TextSuite.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcRelation.scala 35 additions, 1 deletion...ain/scala/org/apache/spark/sql/hive/orc/OrcRelation.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/OrcHadoopFsRelationSuite.scala 28 additions, 1 deletion.../apache/spark/sql/hive/orc/OrcHadoopFsRelationSuite.scala
- sql/hive/src/test/scala/org/apache/spark/sql/sources/ParquetHadoopFsRelationSuite.scala 20 additions, 0 deletions...ache/spark/sql/sources/ParquetHadoopFsRelationSuite.scala
Loading
Please register or sign in to comment