-
- Downloads
Remove the remoteFetchTime metric.
This metric is confusing: it adds up all of the time to fetch shuffle inputs, but fetches often happen in parallel, so remoteFetchTime can be much longer than the task execution time. @squito it looks like you added this metric -- do you have a use case for it? cc @shivaram -- I know you've looked at the shuffle performance a lot so chime in here if this metric has turned out to be useful for you! Author: Kay Ousterhout <kayousterhout@gmail.com> Closes #62 from kayousterhout/remove_fetch_variable and squashes the following commits: 43341eb [Kay Ousterhout] Remote the remoteFetchTime metric.
Showing
- core/src/main/scala/org/apache/spark/BlockStoreShuffleFetcher.scala 0 additions, 1 deletion...ain/scala/org/apache/spark/BlockStoreShuffleFetcher.scala
- core/src/main/scala/org/apache/spark/executor/TaskMetrics.scala 0 additions, 7 deletions...rc/main/scala/org/apache/spark/executor/TaskMetrics.scala
- core/src/main/scala/org/apache/spark/scheduler/JobLogger.scala 0 additions, 1 deletion...src/main/scala/org/apache/spark/scheduler/JobLogger.scala
- core/src/main/scala/org/apache/spark/storage/BlockFetcherIterator.scala 0 additions, 4 deletions...scala/org/apache/spark/storage/BlockFetcherIterator.scala
- core/src/test/scala/org/apache/spark/scheduler/SparkListenerSuite.scala 0 additions, 1 deletion...scala/org/apache/spark/scheduler/SparkListenerSuite.scala
Loading
Please register or sign in to comment