-
- Downloads
[SPARK-6369] [SQL] Uses commit coordinator to help committing Hive and Parquet tables
This PR leverages the output commit coordinator introduced in #4066 to help committing Hive and Parquet tables. This PR extracts output commit code in `SparkHadoopWriter.commit` to `SparkHadoopMapRedUtil.commitTask`, and reuses it for committing Parquet and Hive tables on executor side. TODO - [ ] Add tests <!-- Reviewable:start --> [<img src="https://reviewable.io/review_button.png" height=40 alt="Review on Reviewable"/>](https://reviewable.io/reviews/apache/spark/5139) <!-- Reviewable:end --> Author: Cheng Lian <lian@databricks.com> Closes #5139 from liancheng/spark-6369 and squashes the following commits: 72eb628 [Cheng Lian] Fixes typo in javadoc 9a4b82b [Cheng Lian] Adds javadoc and addresses @aarondav's comments dfdf3ef [Cheng Lian] Uses commit coordinator to help committing Hive and Parquet tables
Showing
- core/src/main/scala/org/apache/spark/SparkHadoopWriter.scala 2 additions, 50 deletionscore/src/main/scala/org/apache/spark/SparkHadoopWriter.scala
- core/src/main/scala/org/apache/spark/mapred/SparkHadoopMapRedUtil.scala 90 additions, 1 deletion...scala/org/apache/spark/mapred/SparkHadoopMapRedUtil.scala
- sql/core/src/main/scala/org/apache/spark/sql/parquet/ParquetTableOperations.scala 6 additions, 5 deletions...org/apache/spark/sql/parquet/ParquetTableOperations.scala
- sql/core/src/main/scala/org/apache/spark/sql/parquet/newParquet.scala 3 additions, 1 deletion.../main/scala/org/apache/spark/sql/parquet/newParquet.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala 0 additions, 1 deletion...apache/spark/sql/hive/execution/InsertIntoHiveTable.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/hiveWriterContainers.scala 2 additions, 15 deletions...cala/org/apache/spark/sql/hive/hiveWriterContainers.scala
Loading
Please register or sign in to comment