-
- Downloads
SPARK-2099. Report progress while task is running.
This is a sketch of a patch that allows the UI to show metrics for tasks that have not yet completed. It adds a heartbeat every 2 seconds from the executors to the driver, reporting metrics for all of the executor's tasks. It still needs unit tests, polish, and cluster testing, but I wanted to put it up to get feedback on the approach. Author: Sandy Ryza <sandy@cloudera.com> Closes #1056 from sryza/sandy-spark-2099 and squashes the following commits: 93b9fdb [Sandy Ryza] Up heartbeat interval to 10 seconds and other tidying 132aec7 [Sandy Ryza] Heartbeat and HeartbeatResponse are already Serializable as case classes 38dffde [Sandy Ryza] Additional review feedback and restore test that was removed in BlockManagerSuite 51fa396 [Sandy Ryza] Remove hostname race, add better comments about threading, and some stylistic improvements 3084f10 [Sandy Ryza] Make TaskUIData a case class again 3bda974 [Sandy Ryza] Stylistic fixes 0dae734 [Sandy Ryza] SPARK-2099. Report progress while task is running.
Showing
- core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala 46 additions, 0 deletionscore/src/main/scala/org/apache/spark/HeartbeatReceiver.scala
- core/src/main/scala/org/apache/spark/SparkContext.scala 4 additions, 0 deletionscore/src/main/scala/org/apache/spark/SparkContext.scala
- core/src/main/scala/org/apache/spark/SparkEnv.scala 1 addition, 7 deletionscore/src/main/scala/org/apache/spark/SparkEnv.scala
- core/src/main/scala/org/apache/spark/executor/Executor.scala 49 additions, 6 deletionscore/src/main/scala/org/apache/spark/executor/Executor.scala
- core/src/main/scala/org/apache/spark/executor/TaskMetrics.scala 9 additions, 1 deletion...rc/main/scala/org/apache/spark/executor/TaskMetrics.scala
- core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala 19 additions, 2 deletions.../main/scala/org/apache/spark/scheduler/DAGScheduler.scala
- core/src/main/scala/org/apache/spark/scheduler/SparkListener.scala 11 additions, 0 deletions...main/scala/org/apache/spark/scheduler/SparkListener.scala
- core/src/main/scala/org/apache/spark/scheduler/SparkListenerBus.scala 2 additions, 0 deletions...n/scala/org/apache/spark/scheduler/SparkListenerBus.scala
- core/src/main/scala/org/apache/spark/scheduler/Task.scala 3 additions, 0 deletionscore/src/main/scala/org/apache/spark/scheduler/Task.scala
- core/src/main/scala/org/apache/spark/scheduler/TaskScheduler.scala 10 additions, 0 deletions...main/scala/org/apache/spark/scheduler/TaskScheduler.scala
- core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala 23 additions, 0 deletions.../scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
- core/src/main/scala/org/apache/spark/scheduler/local/LocalBackend.scala 8 additions, 1 deletion...scala/org/apache/spark/scheduler/local/LocalBackend.scala
- core/src/main/scala/org/apache/spark/storage/BlockManager.scala 1 addition, 24 deletions...rc/main/scala/org/apache/spark/storage/BlockManager.scala
- core/src/main/scala/org/apache/spark/storage/BlockManagerMaster.scala 4 additions, 39 deletions...n/scala/org/apache/spark/storage/BlockManagerMaster.scala
- core/src/main/scala/org/apache/spark/storage/BlockManagerMasterActor.scala 16 additions, 13 deletions...la/org/apache/spark/storage/BlockManagerMasterActor.scala
- core/src/main/scala/org/apache/spark/storage/BlockManagerMessages.scala 3 additions, 3 deletions...scala/org/apache/spark/storage/BlockManagerMessages.scala
- core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala 74 additions, 43 deletions.../scala/org/apache/spark/ui/jobs/JobProgressListener.scala
- core/src/main/scala/org/apache/spark/ui/jobs/UIData.scala 6 additions, 3 deletionscore/src/main/scala/org/apache/spark/ui/jobs/UIData.scala
- core/src/main/scala/org/apache/spark/util/AkkaUtils.scala 64 additions, 2 deletionscore/src/main/scala/org/apache/spark/util/AkkaUtils.scala
- core/src/test/scala/org/apache/spark/SparkContextSchedulerCreationSuite.scala 3 additions, 3 deletions...org/apache/spark/SparkContextSchedulerCreationSuite.scala
Loading
Please register or sign in to comment