-
- Downloads
[SPARK-19820][CORE] Add interface to kill tasks w/ a reason
This commit adds a killTaskAttempt method to SparkContext, to allow users to kill tasks so that they can be re-scheduled elsewhere. This also refactors the task kill path to allow specifying a reason for the task kill. The reason is propagated opaquely through events, and will show up in the UI automatically as `(N killed: $reason)` and `TaskKilled: $reason`. Without this change, there is no way to provide the user feedback through the UI. Currently used reasons are "stage cancelled", "another attempt succeeded", and "killed via SparkContext.killTask". The user can also specify a custom reason through `SparkContext.killTask`. cc rxin In the stage overview UI the reasons are summarized:  Within the stage UI you can see individual task kill reasons:  Existing tests, tried killing some stages in the UI and verified the messages are as expected. Author: Eric Liang <ekl@databricks.com> Author: Eric Liang <ekl@google.com> Closes #17166 from ericl/kill-reason.
Showing
- core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeInMemorySorter.java 2 additions, 3 deletions...ark/util/collection/unsafe/sort/UnsafeInMemorySorter.java
- core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeSorterSpillReader.java 2 additions, 3 deletions.../util/collection/unsafe/sort/UnsafeSorterSpillReader.java
- core/src/main/scala/org/apache/spark/InterruptibleIterator.scala 2 additions, 5 deletions...c/main/scala/org/apache/spark/InterruptibleIterator.scala
- core/src/main/scala/org/apache/spark/SparkContext.scala 18 additions, 0 deletionscore/src/main/scala/org/apache/spark/SparkContext.scala
- core/src/main/scala/org/apache/spark/TaskContext.scala 10 additions, 0 deletionscore/src/main/scala/org/apache/spark/TaskContext.scala
- core/src/main/scala/org/apache/spark/TaskContextImpl.scala 16 additions, 5 deletionscore/src/main/scala/org/apache/spark/TaskContextImpl.scala
- core/src/main/scala/org/apache/spark/TaskEndReason.scala 2 additions, 2 deletionscore/src/main/scala/org/apache/spark/TaskEndReason.scala
- core/src/main/scala/org/apache/spark/TaskKilledException.scala 3 additions, 1 deletion...src/main/scala/org/apache/spark/TaskKilledException.scala
- core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala 1 addition, 1 deletion...rc/main/scala/org/apache/spark/api/python/PythonRDD.scala
- core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala 2 additions, 2 deletions.../apache/spark/executor/CoarseGrainedExecutorBackend.scala
- core/src/main/scala/org/apache/spark/executor/Executor.scala 28 additions, 24 deletionscore/src/main/scala/org/apache/spark/executor/Executor.scala
- core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala 10 additions, 1 deletion.../main/scala/org/apache/spark/scheduler/DAGScheduler.scala
- core/src/main/scala/org/apache/spark/scheduler/SchedulerBackend.scala 14 additions, 1 deletion...n/scala/org/apache/spark/scheduler/SchedulerBackend.scala
- core/src/main/scala/org/apache/spark/scheduler/Task.scala 11 additions, 10 deletionscore/src/main/scala/org/apache/spark/scheduler/Task.scala
- core/src/main/scala/org/apache/spark/scheduler/TaskScheduler.scala 7 additions, 0 deletions...main/scala/org/apache/spark/scheduler/TaskScheduler.scala
- core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala 14 additions, 2 deletions.../scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
- core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala 9 additions, 1 deletion...ain/scala/org/apache/spark/scheduler/TaskSetManager.scala
- core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedClusterMessage.scala 1 addition, 1 deletion...spark/scheduler/cluster/CoarseGrainedClusterMessage.scala
- core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala 6 additions, 4 deletions...ark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
- core/src/main/scala/org/apache/spark/scheduler/local/LocalSchedulerBackend.scala 6 additions, 5 deletions.../apache/spark/scheduler/local/LocalSchedulerBackend.scala
Loading
Please register or sign in to comment