-
- Downloads
[SPARK-18024][SQL] Introduce an internal commit protocol API
## What changes were proposed in this pull request? This patch introduces an internal commit protocol API that is used by the batch data source to do write commits. It currently has only one implementation that uses Hadoop MapReduce's OutputCommitter API. In the future, this commit API can be used to unify streaming and batch commits. ## How was this patch tested? Should be covered by existing write tests. Author: Reynold Xin <rxin@databricks.com> Author: Eric Liang <ekl@databricks.com> Closes #15707 from rxin/SPARK-18024-2.
Showing
- mllib/src/main/scala/org/apache/spark/ml/source/libsvm/LibSVMRelation.scala 7 additions, 10 deletions...la/org/apache/spark/ml/source/libsvm/LibSVMRelation.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileCommitProtocol.scala 254 additions, 0 deletions.../spark/sql/execution/datasources/FileCommitProtocol.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/OutputWriter.scala 6 additions, 20 deletions...apache/spark/sql/execution/datasources/OutputWriter.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/WriteOutput.scala 43 additions, 124 deletions.../apache/spark/sql/execution/datasources/WriteOutput.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVRelation.scala 7 additions, 10 deletions...che/spark/sql/execution/datasources/csv/CSVRelation.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonFileFormat.scala 7 additions, 10 deletions...spark/sql/execution/datasources/json/JsonFileFormat.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala 6 additions, 2 deletions...sql/execution/datasources/parquet/ParquetFileFormat.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetOutputWriter.scala 6 additions, 13 deletions...l/execution/datasources/parquet/ParquetOutputWriter.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/text/TextFileFormat.scala 7 additions, 10 deletions...spark/sql/execution/datasources/text/TextFileFormat.scala
- sql/core/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala 20 additions, 9 deletions...rc/main/scala/org/apache/spark/sql/internal/SQLConf.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFileFormat.scala 12 additions, 16 deletions...n/scala/org/apache/spark/sql/hive/orc/OrcFileFormat.scala
- sql/hive/src/test/scala/org/apache/spark/sql/sources/CommitFailureTestSource.scala 4 additions, 6 deletions...rg/apache/spark/sql/sources/CommitFailureTestSource.scala
- sql/hive/src/test/scala/org/apache/spark/sql/sources/SimpleTextRelation.scala 8 additions, 11 deletions...ala/org/apache/spark/sql/sources/SimpleTextRelation.scala
Loading
Please register or sign in to comment