-
- Downloads
[SPARK-4092] [CORE] Fix InputMetrics for coalesce'd Rdds
When calculating the input metrics there was an assumption that one task only reads from one block - this is not true for some operations including coalesce. This patch simply increments the task's input metrics if previous ones existed of the same read method. A limitation to this patch is that if a task reads from two different blocks of different read methods, one will override the other. Author: Kostas Sakellis <kostas@cloudera.com> Closes #3120 from ksakellis/kostas-spark-4092 and squashes the following commits: 54e6658 [Kostas Sakellis] Drops metrics if conflicting read methods exist f0e0cc5 [Kostas Sakellis] Add bytesReadCallback to InputMetrics a2a36d4 [Kostas Sakellis] CR feedback 5a0c770 [Kostas Sakellis] [SPARK-4092] [CORE] Fix InputMetrics for coalesce'd Rdds
Showing
- core/src/main/scala/org/apache/spark/CacheManager.scala 5 additions, 1 deletioncore/src/main/scala/org/apache/spark/CacheManager.scala
- core/src/main/scala/org/apache/spark/executor/Executor.scala 1 addition, 0 deletionscore/src/main/scala/org/apache/spark/executor/Executor.scala
- core/src/main/scala/org/apache/spark/executor/TaskMetrics.scala 73 additions, 2 deletions...rc/main/scala/org/apache/spark/executor/TaskMetrics.scala
- core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala 13 additions, 26 deletionscore/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala
- core/src/main/scala/org/apache/spark/rdd/NewHadoopRDD.scala 13 additions, 27 deletionscore/src/main/scala/org/apache/spark/rdd/NewHadoopRDD.scala
- core/src/main/scala/org/apache/spark/storage/BlockManager.scala 1 addition, 1 deletion...rc/main/scala/org/apache/spark/storage/BlockManager.scala
- core/src/main/scala/org/apache/spark/util/JsonProtocol.scala 3 additions, 3 deletionscore/src/main/scala/org/apache/spark/util/JsonProtocol.scala
- core/src/test/scala/org/apache/spark/metrics/InputOutputMetricsSuite.scala 157 additions, 38 deletions...la/org/apache/spark/metrics/InputOutputMetricsSuite.scala
- core/src/test/scala/org/apache/spark/ui/jobs/JobProgressListenerSuite.scala 2 additions, 2 deletions...a/org/apache/spark/ui/jobs/JobProgressListenerSuite.scala
- core/src/test/scala/org/apache/spark/util/JsonProtocolSuite.scala 2 additions, 2 deletions.../test/scala/org/apache/spark/util/JsonProtocolSuite.scala
Loading
Please register or sign in to comment