-
- Downloads
[SPARK-21621][CORE] Reset numRecordsWritten after DiskBlockObjectWriter.commitAndGet called
## What changes were proposed in this pull request? We should reset numRecordsWritten to zero after DiskBlockObjectWriter.commitAndGet called. Because when `revertPartialWritesAndClose` be called, we decrease the written records in `ShuffleWriteMetrics` . However, we decreased the written records to zero, this should be wrong, we should only decreased the number reords after the last `commitAndGet` called. ## How was this patch tested? Modified existing test. Please review http://spark.apache.org/contributing.html before opening a pull request. Author: Xianyang Liu <xianyang.liu@intel.com> Closes #18830 from ConeyLiu/DiskBlockObjectWriter.
Showing
- core/src/main/scala/org/apache/spark/storage/DiskBlockObjectWriter.scala 2 additions, 0 deletions...cala/org/apache/spark/storage/DiskBlockObjectWriter.scala
- core/src/test/scala/org/apache/spark/storage/DiskBlockObjectWriterSuite.scala 1 addition, 0 deletions...org/apache/spark/storage/DiskBlockObjectWriterSuite.scala
Loading
Please register or sign in to comment