-
- Downloads
[SPARK-6352] [SQL] Add DirectParquetOutputCommitter
Add a DirectParquetOutputCommitter class that skips _temporary directory when saving to s3. Add new config value "spark.sql.parquet.useDirectParquetOutputCommitter" (default false) to choose between the default output committer. Author: Pei-Lun Lee <pllee@appier.com> Closes #5042 from ypcat/spark-6352 and squashes the following commits: e17bf47 [Pei-Lun Lee] Merge branch 'master' of https://github.com/apache/spark into spark-6352 9ae7545 [Pei-Lun Lee] [SPARL-6352] [SQL] Change to allow custom parquet output committer. 0d540b9 [Pei-Lun Lee] [SPARK-6352] [SQL] add license c42468c [Pei-Lun Lee] [SPARK-6352] [SQL] add test case 0fc03ca [Pei-Lun Lee] [SPARK-6532] [SQL] hide class DirectParquetOutputCommitter 769bd67 [Pei-Lun Lee] DirectParquetOutputCommitter f75e261 [Pei-Lun Lee] DirectParquetOutputCommitter
Showing
- sql/core/src/main/scala/org/apache/spark/sql/parquet/DirectParquetOutputCommitter.scala 66 additions, 0 deletions...ache/spark/sql/parquet/DirectParquetOutputCommitter.scala
- sql/core/src/main/scala/org/apache/spark/sql/parquet/ParquetTableOperations.scala 22 additions, 0 deletions...org/apache/spark/sql/parquet/ParquetTableOperations.scala
- sql/core/src/test/scala/org/apache/spark/sql/parquet/ParquetIOSuite.scala 21 additions, 0 deletions...t/scala/org/apache/spark/sql/parquet/ParquetIOSuite.scala
Loading
Please register or sign in to comment