-
- Downloads
[SPARK-611] Display executor thread dumps in web UI
This patch allows executor thread dumps to be collected on-demand and viewed in the Spark web UI. The thread dumps are collected using Thread.getAllStackTraces(). To allow remote thread dumps to be triggered from the web UI, I added a new `ExecutorActor` that runs inside of the Executor actor system and responds to RPCs from the driver. The driver's mechanism for obtaining a reference to this actor is a little bit hacky: it uses the block manager master actor to determine the host/port of the executor actor systems in order to construct ActorRefs to ExecutorActor. Unfortunately, I couldn't find a much cleaner way to do this without a big refactoring of the executor -> driver communication. Screenshots:    Author: Josh Rosen <joshrosen@databricks.com> Closes #2944 from JoshRosen/jstack-in-web-ui and squashes the following commits: 3c21a5d [Josh Rosen] Address review comments: 880f7f7 [Josh Rosen] Merge remote-tracking branch 'origin/master' into jstack-in-web-ui f719266 [Josh Rosen] Merge remote-tracking branch 'origin/master' into jstack-in-web-ui 19707b0 [Josh Rosen] Add one comment. 127a130 [Josh Rosen] Update to use SparkContext.DRIVER_IDENTIFIER b8e69aa [Josh Rosen] Merge remote-tracking branch 'origin/master' into jstack-in-web-ui 3dfc2d4 [Josh Rosen] Add missing file. bc1e675 [Josh Rosen] Undo some leftover changes from the earlier approach. f4ac1c1 [Josh Rosen] Switch to on-demand collection of thread dumps dfec08b [Josh Rosen] Add option to disable thread dumps in UI. 4c87d7f [Josh Rosen] Use separate RPC for sending thread dumps. 2b8bdf3 [Josh Rosen] Enable thread dumps from the driver when running in non-local mode. cc3e6b3 [Josh Rosen] Fix test code in DAGSchedulerSuite. 87b8b65 [Josh Rosen] Add new listener event for thread dumps. 8c10216 [Josh Rosen] Add missing file. 0f198ac [Josh Rosen] [SPARK-611] Display executor thread dumps in web UI
Showing
- core/src/main/scala/org/apache/spark/SparkContext.scala 26 additions, 3 deletionscore/src/main/scala/org/apache/spark/SparkContext.scala
- core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala 2 additions, 1 deletion.../apache/spark/executor/CoarseGrainedExecutorBackend.scala
- core/src/main/scala/org/apache/spark/executor/Executor.scala 6 additions, 1 deletioncore/src/main/scala/org/apache/spark/executor/Executor.scala
- core/src/main/scala/org/apache/spark/executor/ExecutorActor.scala 41 additions, 0 deletions.../main/scala/org/apache/spark/executor/ExecutorActor.scala
- core/src/main/scala/org/apache/spark/storage/BlockManagerMaster.scala 4 additions, 0 deletions...n/scala/org/apache/spark/storage/BlockManagerMaster.scala
- core/src/main/scala/org/apache/spark/storage/BlockManagerMasterActor.scala 18 additions, 0 deletions...la/org/apache/spark/storage/BlockManagerMasterActor.scala
- core/src/main/scala/org/apache/spark/storage/BlockManagerMessages.scala 2 additions, 0 deletions...scala/org/apache/spark/storage/BlockManagerMessages.scala
- core/src/main/scala/org/apache/spark/ui/exec/ExecutorThreadDumpPage.scala 73 additions, 0 deletions...ala/org/apache/spark/ui/exec/ExecutorThreadDumpPage.scala
- core/src/main/scala/org/apache/spark/ui/exec/ExecutorsPage.scala 14 additions, 1 deletion...c/main/scala/org/apache/spark/ui/exec/ExecutorsPage.scala
- core/src/main/scala/org/apache/spark/ui/exec/ExecutorsTab.scala 7 additions, 1 deletion...rc/main/scala/org/apache/spark/ui/exec/ExecutorsTab.scala
- core/src/main/scala/org/apache/spark/util/AkkaUtils.scala 14 additions, 0 deletionscore/src/main/scala/org/apache/spark/util/AkkaUtils.scala
- core/src/main/scala/org/apache/spark/util/ThreadStackTrace.scala 27 additions, 0 deletions...c/main/scala/org/apache/spark/util/ThreadStackTrace.scala
- core/src/main/scala/org/apache/spark/util/Utils.scala 13 additions, 0 deletionscore/src/main/scala/org/apache/spark/util/Utils.scala
Loading
Please register or sign in to comment