-
- Downloads
SPARK-2566. Update ShuffleWriteMetrics incrementally
I haven't tested this out on a cluster yet, but wanted to make sure the approach (passing ShuffleWriteMetrics down to DiskBlockObjectWriter) was ok Author: Sandy Ryza <sandy@cloudera.com> Closes #1481 from sryza/sandy-spark-2566 and squashes the following commits: 8090d88 [Sandy Ryza] Fix ExternalSorter b2a62ed [Sandy Ryza] Fix more test failures 8be6218 [Sandy Ryza] Fix test failures and mark a couple variables private c5e68e5 [Sandy Ryza] SPARK-2566. Update ShuffleWriteMetrics incrementally
Showing
- core/src/main/scala/org/apache/spark/executor/TaskMetrics.scala 2 additions, 2 deletions...rc/main/scala/org/apache/spark/executor/TaskMetrics.scala
- core/src/main/scala/org/apache/spark/shuffle/hash/HashShuffleWriter.scala 5 additions, 11 deletions...ala/org/apache/spark/shuffle/hash/HashShuffleWriter.scala
- core/src/main/scala/org/apache/spark/shuffle/sort/SortShuffleWriter.scala 5 additions, 11 deletions...ala/org/apache/spark/shuffle/sort/SortShuffleWriter.scala
- core/src/main/scala/org/apache/spark/storage/BlockManager.scala 7 additions, 5 deletions...rc/main/scala/org/apache/spark/storage/BlockManager.scala
- core/src/main/scala/org/apache/spark/storage/BlockObjectWriter.scala 42 additions, 35 deletions...in/scala/org/apache/spark/storage/BlockObjectWriter.scala
- core/src/main/scala/org/apache/spark/storage/ShuffleBlockManager.scala 6 additions, 3 deletions.../scala/org/apache/spark/storage/ShuffleBlockManager.scala
- core/src/main/scala/org/apache/spark/util/collection/ExternalAppendOnlyMap.scala 13 additions, 5 deletions.../apache/spark/util/collection/ExternalAppendOnlyMap.scala
- core/src/main/scala/org/apache/spark/util/collection/ExternalSorter.scala 11 additions, 6 deletions...ala/org/apache/spark/util/collection/ExternalSorter.scala
- core/src/test/scala/org/apache/spark/storage/BlockObjectWriterSuite.scala 65 additions, 0 deletions...ala/org/apache/spark/storage/BlockObjectWriterSuite.scala
- core/src/test/scala/org/apache/spark/storage/DiskBlockManagerSuite.scala 6 additions, 3 deletions...cala/org/apache/spark/storage/DiskBlockManagerSuite.scala
- tools/src/main/scala/org/apache/spark/tools/StoragePerfTester.scala 2 additions, 1 deletion...main/scala/org/apache/spark/tools/StoragePerfTester.scala
Loading
Please register or sign in to comment