-
- Downloads
SPARK-2035: Store call stack for stages, display it on the UI.
I'm not sure about the test -- I get a lot of unrelated failures for some reason. I'll try to sort it out. But hopefully the automation will test this for me if I send a pull request :). I'll attach a demo HTML in [Jira](https://issues.apache.org/jira/browse/SPARK-2035). Author: Daniel Darabos <darabos.daniel@gmail.com> Author: Patrick Wendell <pwendell@gmail.com> Closes #981 from darabos/darabos-call-stack and squashes the following commits: f7c6bfa [Daniel Darabos] Fix bad merge. I undid 83c226d4 by Doris. 3d0a48d [Daniel Darabos] Merge remote-tracking branch 'upstream/master' into darabos-call-stack b857849 [Daniel Darabos] Style: Break long line. ecb5690 [Daniel Darabos] Include the last Spark method in the full stack trace. Otherwise it is not visible if the stage name is overridden. d00a85b [Patrick Wendell] Make call sites for stages non-optional and well defined b9eba24 [Daniel Darabos] Make StageInfo.details non-optional. Add JSON serialization code for the new field. Verify JSON backward compatibility. 4312828 [Daniel Darabos] Remove Mima excludes for CallSite. They should be unnecessary now, with SPARK-2070 fixed. 0920750 [Daniel Darabos] Merge remote-tracking branch 'upstream/master' into darabos-call-stack a4b1faf [Daniel Darabos] Add Mima exclusions for the CallSite changes it has picked up. They are private methods/classes, so we ought to be safe. 932f810 [Daniel Darabos] Use empty CallSite instead of null in DAGSchedulerSuite. Outside of testing, this parameter always originates in SparkContext.scala, and will never be null. ccd89d1 [Daniel Darabos] Fix long lines. ac173e4 [Daniel Darabos] Hide "show details" if there are no details to show. 6182da6 [Daniel Darabos] Set a configurable limit on maximum call stack depth. It can be useful in memory-constrained situations with large numbers of stages. 8fe2e34 [Daniel Darabos] Store call stack for stages, display it on the UI.
Showing
- core/src/main/resources/org/apache/spark/ui/static/webui.css 21 additions, 0 deletionscore/src/main/resources/org/apache/spark/ui/static/webui.css
- core/src/main/scala/org/apache/spark/SparkContext.scala 10 additions, 8 deletionscore/src/main/scala/org/apache/spark/SparkContext.scala
- core/src/main/scala/org/apache/spark/rdd/RDD.scala 3 additions, 3 deletionscore/src/main/scala/org/apache/spark/rdd/RDD.scala
- core/src/main/scala/org/apache/spark/scheduler/ActiveJob.scala 2 additions, 1 deletion...src/main/scala/org/apache/spark/scheduler/ActiveJob.scala
- core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala 13 additions, 11 deletions.../main/scala/org/apache/spark/scheduler/DAGScheduler.scala
- core/src/main/scala/org/apache/spark/scheduler/DAGSchedulerEvent.scala 2 additions, 1 deletion.../scala/org/apache/spark/scheduler/DAGSchedulerEvent.scala
- core/src/main/scala/org/apache/spark/scheduler/Stage.scala 9 additions, 2 deletionscore/src/main/scala/org/apache/spark/scheduler/Stage.scala
- core/src/main/scala/org/apache/spark/scheduler/StageInfo.scala 7 additions, 2 deletions...src/main/scala/org/apache/spark/scheduler/StageInfo.scala
- core/src/main/scala/org/apache/spark/ui/jobs/StageTable.scala 9 additions, 1 deletion.../src/main/scala/org/apache/spark/ui/jobs/StageTable.scala
- core/src/main/scala/org/apache/spark/util/JsonProtocol.scala 3 additions, 1 deletioncore/src/main/scala/org/apache/spark/util/JsonProtocol.scala
- core/src/main/scala/org/apache/spark/util/Utils.scala 16 additions, 16 deletionscore/src/main/scala/org/apache/spark/util/Utils.scala
- core/src/test/scala/org/apache/spark/SparkContextInfoSuite.scala 1 addition, 1 deletion...c/test/scala/org/apache/spark/SparkContextInfoSuite.scala
- core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala 4 additions, 3 deletions.../scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
- core/src/test/scala/org/apache/spark/ui/jobs/JobProgressListenerSuite.scala 2 additions, 2 deletions...a/org/apache/spark/ui/jobs/JobProgressListenerSuite.scala
- core/src/test/scala/org/apache/spark/util/JsonProtocolSuite.scala 13 additions, 1 deletion.../test/scala/org/apache/spark/util/JsonProtocolSuite.scala
Loading
Please register or sign in to comment