-
- Downloads
[SPARK-18219] Move commit protocol API (internal) from sql/core to core module
## What changes were proposed in this pull request? This patch moves the new commit protocol API from sql/core to core module, so we can use it in the future in the RDD API. As part of this patch, I also moved the speficiation of the random uuid for the write path out of the commit protocol, and instead pass in a job id. ## How was this patch tested? N/A Author: Reynold Xin <rxin@databricks.com> Closes #15731 from rxin/SPARK-18219. (cherry picked from commit 937af592) Signed-off-by:Reynold Xin <rxin@databricks.com>
Showing
- core/src/main/scala/org/apache/spark/internal/io/FileCommitProtocol.scala 126 additions, 0 deletions...ala/org/apache/spark/internal/io/FileCommitProtocol.scala
- core/src/main/scala/org/apache/spark/internal/io/HadoopMapReduceCommitProtocol.scala 111 additions, 0 deletions...che/spark/internal/io/HadoopMapReduceCommitProtocol.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatWriter.scala 2 additions, 1 deletion...he/spark/sql/execution/datasources/FileFormatWriter.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/InsertIntoHadoopFsRelationCommand.scala 4 additions, 2 deletions...ution/datasources/InsertIntoHadoopFsRelationCommand.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/SQLHadoopMapReduceCommitProtocol.scala 72 additions, 0 deletions...cution/datasources/SQLHadoopMapReduceCommitProtocol.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSink.scala 7 additions, 2 deletions...apache/spark/sql/execution/streaming/FileStreamSink.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/ManifestFileCommitProtocol.scala 3 additions, 3 deletions.../sql/execution/streaming/ManifestFileCommitProtocol.scala
- sql/core/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala 2 additions, 2 deletions...rc/main/scala/org/apache/spark/sql/internal/SQLConf.scala
Loading
Please register or sign in to comment