-
- Downloads
[SPARK-4874] [CORE] Collect record count metrics
Collects record counts for both Input/Output and Shuffle Metrics. For the input/output metrics, it just appends the counter every time the iterators get accessed. For shuffle on the write side, we count the metrics post aggregation (after a map side combine) and on the read side we count the metrics pre aggregation. This allows both the bytes read/written metrics and the records read/written to line up. For backwards compatibility, if we deserialize an older event that doesn't have record metrics, we set the metric to -1. Author: Kostas Sakellis <kostas@cloudera.com> Closes #4067 from ksakellis/kostas-spark-4874 and squashes the following commits: bd919be [Kostas Sakellis] Changed 'Records Read' in shuffleReadMetrics json output to 'Total Records Read' dad4d57 [Kostas Sakellis] Add a comment and check to BlockObjectWriter so that it cannot be reopend. 6f236a1 [Kostas Sakellis] Renamed _recordsWritten in ShuffleWriteMetrics to be more consistent 70620a0 [Kostas Sakellis] CR Feedback 17faa3a [Kostas Sakellis] Removed AtomicLong in favour of using Long b6f9923 [Kostas Sakellis] Merge AfterNextInterceptingIterator with InterruptableIterator to save a function call 46c8186 [Kostas Sakellis] Combined Bytes and # records into one column 57551c1 [Kostas Sakellis] Conforms to SPARK-3288 6cdb44e [Kostas Sakellis] Removed the generic InterceptingIterator and repalced it with specific implementation 1aa273c [Kostas Sakellis] CR Feedback 1bb78b1 [Kostas Sakellis] [SPARK-4874] [CORE] Collect record count metrics
Showing
- core/src/main/scala/org/apache/spark/CacheManager.scala 8 additions, 3 deletionscore/src/main/scala/org/apache/spark/CacheManager.scala
- core/src/main/scala/org/apache/spark/executor/TaskMetrics.scala 40 additions, 14 deletions...rc/main/scala/org/apache/spark/executor/TaskMetrics.scala
- core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala 4 additions, 2 deletionscore/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala
- core/src/main/scala/org/apache/spark/rdd/NewHadoopRDD.scala 4 additions, 2 deletionscore/src/main/scala/org/apache/spark/rdd/NewHadoopRDD.scala
- core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala 7 additions, 5 deletions...rc/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala
- core/src/main/scala/org/apache/spark/shuffle/hash/BlockStoreShuffleFetcher.scala 7 additions, 1 deletion.../apache/spark/shuffle/hash/BlockStoreShuffleFetcher.scala
- core/src/main/scala/org/apache/spark/storage/BlockManager.scala 1 addition, 1 deletion...rc/main/scala/org/apache/spark/storage/BlockManager.scala
- core/src/main/scala/org/apache/spark/storage/BlockObjectWriter.scala 16 additions, 8 deletions...in/scala/org/apache/spark/storage/BlockObjectWriter.scala
- core/src/main/scala/org/apache/spark/ui/ToolTips.scala 5 additions, 4 deletionscore/src/main/scala/org/apache/spark/ui/ToolTips.scala
- core/src/main/scala/org/apache/spark/ui/exec/ExecutorsTab.scala 6 additions, 0 deletions...rc/main/scala/org/apache/spark/ui/exec/ExecutorsTab.scala
- core/src/main/scala/org/apache/spark/ui/jobs/ExecutorTable.scala 68 additions, 18 deletions...c/main/scala/org/apache/spark/ui/jobs/ExecutorTable.scala
- core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala 24 additions, 0 deletions.../scala/org/apache/spark/ui/jobs/JobProgressListener.scala
- core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala 95 additions, 53 deletionscore/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala
- core/src/main/scala/org/apache/spark/ui/jobs/UIData.scala 14 additions, 0 deletionscore/src/main/scala/org/apache/spark/ui/jobs/UIData.scala
- core/src/main/scala/org/apache/spark/util/JsonProtocol.scala 14 additions, 5 deletionscore/src/main/scala/org/apache/spark/util/JsonProtocol.scala
- core/src/main/scala/org/apache/spark/util/collection/ExternalSorter.scala 1 addition, 0 deletions...ala/org/apache/spark/util/collection/ExternalSorter.scala
- core/src/test/scala/org/apache/spark/metrics/InputOutputMetricsSuite.scala 165 additions, 21 deletions...la/org/apache/spark/metrics/InputOutputMetricsSuite.scala
- core/src/test/scala/org/apache/spark/storage/BlockObjectWriterSuite.scala 21 additions, 0 deletions...ala/org/apache/spark/storage/BlockObjectWriterSuite.scala
- core/src/test/scala/org/apache/spark/ui/jobs/JobProgressListenerSuite.scala 1 addition, 1 deletion...a/org/apache/spark/ui/jobs/JobProgressListenerSuite.scala
- core/src/test/scala/org/apache/spark/util/JsonProtocolSuite.scala 47 additions, 8 deletions.../test/scala/org/apache/spark/util/JsonProtocolSuite.scala
Loading
Please register or sign in to comment