-
- Downloads
[SPARK-2298] Encode stage attempt in SparkListener & UI.
Simple way to reproduce this in the UI: ```scala val f = new java.io.File("/tmp/test") f.delete() sc.parallelize(1 to 2, 2).map(x => (x,x )).repartition(3).mapPartitionsWithContext { case (context, iter) => if (context.partitionId == 0) { val f = new java.io.File("/tmp/test") if (!f.exists) { f.mkdir() System.exit(0); } } iter }.count() ``` Author: Reynold Xin <rxin@apache.org> Closes #1545 from rxin/stage-attempt and squashes the following commits: 3ee1d2a [Reynold Xin] - Rename attempt to retry in UI. - Properly report stage failure in FetchFailed. 40a6bd5 [Reynold Xin] Updated test suites. c414c36 [Reynold Xin] Fixed the hanging in JobCancellationSuite. b3e2eed [Reynold Xin] Oops previous code didn't compile. 0f36075 [Reynold Xin] Mark unknown stage attempt with id -1 and drop that in JobProgressListener. 6c08b07 [Reynold Xin] Addressed code review feedback. 4e5faa2 [Reynold Xin] [SPARK-2298] Encode stage attempt in SparkListener & UI.
Showing
- core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala 48 additions, 29 deletions.../main/scala/org/apache/spark/scheduler/DAGScheduler.scala
- core/src/main/scala/org/apache/spark/scheduler/SparkListener.scala 9 additions, 2 deletions...main/scala/org/apache/spark/scheduler/SparkListener.scala
- core/src/main/scala/org/apache/spark/scheduler/Stage.scala 6 additions, 2 deletionscore/src/main/scala/org/apache/spark/scheduler/Stage.scala
- core/src/main/scala/org/apache/spark/scheduler/StageInfo.scala 9 additions, 2 deletions...src/main/scala/org/apache/spark/scheduler/StageInfo.scala
- core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala 4 additions, 4 deletions.../scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
- core/src/main/scala/org/apache/spark/scheduler/TaskSet.scala 0 additions, 4 deletionscore/src/main/scala/org/apache/spark/scheduler/TaskSet.scala
- core/src/main/scala/org/apache/spark/ui/jobs/ExecutorTable.scala 3 additions, 3 deletions...c/main/scala/org/apache/spark/ui/jobs/ExecutorTable.scala
- core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala 24 additions, 16 deletions.../scala/org/apache/spark/ui/jobs/JobProgressListener.scala
- core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala 7 additions, 4 deletionscore/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala
- core/src/main/scala/org/apache/spark/ui/jobs/StageTable.scala 9 additions, 5 deletions.../src/main/scala/org/apache/spark/ui/jobs/StageTable.scala
- core/src/main/scala/org/apache/spark/util/JsonProtocol.scala 9 additions, 3 deletionscore/src/main/scala/org/apache/spark/util/JsonProtocol.scala
- core/src/test/scala/org/apache/spark/storage/StorageStatusListenerSuite.scala 9 additions, 8 deletions...org/apache/spark/storage/StorageStatusListenerSuite.scala
- core/src/test/scala/org/apache/spark/ui/jobs/JobProgressListenerSuite.scala 37 additions, 31 deletions...a/org/apache/spark/ui/jobs/JobProgressListenerSuite.scala
- core/src/test/scala/org/apache/spark/ui/storage/StorageTabSuite.scala 8 additions, 8 deletions...t/scala/org/apache/spark/ui/storage/StorageTabSuite.scala
- core/src/test/scala/org/apache/spark/util/JsonProtocolSuite.scala 373 additions, 103 deletions.../test/scala/org/apache/spark/util/JsonProtocolSuite.scala
Loading
Please register or sign in to comment