-
- Downloads
[SPARK-10530][CORE] Kill other task attempts when one taskattempt belonging...
[SPARK-10530][CORE] Kill other task attempts when one taskattempt belonging the same task is succeeded in speculation ## What changes were proposed in this pull request? With this patch, TaskSetManager kills other running attempts when any one of the attempt succeeds for the same task. Also killed tasks will not be considered as failed tasks and they get listed separately in the UI and also shows the task state as KILLED instead of FAILED. ## How was this patch tested? core\src\test\scala\org\apache\spark\ui\jobs\JobProgressListenerSuite.scala core\src\test\scala\org\apache\spark\util\JsonProtocolSuite.scala I have verified this patch manually by enabling spark.speculation as true, when any attempt gets succeeded then other running attempts are getting killed for the same task and other pending tasks are getting assigned in those. And also when any attempt gets killed then they are considered as KILLED tasks and not considered as FAILED tasks. Please find the attached screen shots for the reference.   Ref : https://github.com/apache/spark/pull/11916 Author: Devaraj K <devaraj@apache.org> Closes #11996 from devaraj-kavali/SPARK-10530.
Showing
- core/src/main/scala/org/apache/spark/scheduler/Pool.scala 2 additions, 2 deletionscore/src/main/scala/org/apache/spark/scheduler/Pool.scala
- core/src/main/scala/org/apache/spark/scheduler/Schedulable.scala 1 addition, 1 deletion...c/main/scala/org/apache/spark/scheduler/Schedulable.scala
- core/src/main/scala/org/apache/spark/scheduler/TaskInfo.scala 13 additions, 7 deletions.../src/main/scala/org/apache/spark/scheduler/TaskInfo.scala
- core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala 6 additions, 1 deletion.../scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
- core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala 13 additions, 5 deletions...ain/scala/org/apache/spark/scheduler/TaskSetManager.scala
- core/src/main/scala/org/apache/spark/ui/UIUtils.scala 2 additions, 0 deletionscore/src/main/scala/org/apache/spark/ui/UIUtils.scala
- core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala 1 addition, 1 deletion...src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala
- core/src/main/scala/org/apache/spark/ui/jobs/ExecutorTable.scala 3 additions, 1 deletion...c/main/scala/org/apache/spark/ui/jobs/ExecutorTable.scala
- core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala 7 additions, 0 deletions.../scala/org/apache/spark/ui/jobs/JobProgressListener.scala
- core/src/main/scala/org/apache/spark/ui/jobs/StageTable.scala 1 addition, 1 deletion.../src/main/scala/org/apache/spark/ui/jobs/StageTable.scala
- core/src/main/scala/org/apache/spark/ui/jobs/UIData.scala 3 additions, 0 deletionscore/src/main/scala/org/apache/spark/ui/jobs/UIData.scala
- core/src/main/scala/org/apache/spark/util/JsonProtocol.scala 3 additions, 0 deletionscore/src/main/scala/org/apache/spark/util/JsonProtocol.scala
- core/src/test/scala/org/apache/spark/scheduler/TaskSetManagerSuite.scala 50 additions, 0 deletions...cala/org/apache/spark/scheduler/TaskSetManagerSuite.scala
- core/src/test/scala/org/apache/spark/ui/StagePageSuite.scala 1 addition, 1 deletioncore/src/test/scala/org/apache/spark/ui/StagePageSuite.scala
- core/src/test/scala/org/apache/spark/ui/UIUtilsSuite.scala 1 addition, 1 deletioncore/src/test/scala/org/apache/spark/ui/UIUtilsSuite.scala
- core/src/test/scala/org/apache/spark/ui/jobs/JobProgressListenerSuite.scala 5 additions, 1 deletion...a/org/apache/spark/ui/jobs/JobProgressListenerSuite.scala
- core/src/test/scala/org/apache/spark/util/JsonProtocolSuite.scala 5 additions, 0 deletions.../test/scala/org/apache/spark/util/JsonProtocolSuite.scala
- streaming/src/main/scala/org/apache/spark/streaming/ui/AllBatchesTable.scala 1 addition, 0 deletions...scala/org/apache/spark/streaming/ui/AllBatchesTable.scala
- streaming/src/main/scala/org/apache/spark/streaming/ui/BatchPage.scala 1 addition, 0 deletions.../main/scala/org/apache/spark/streaming/ui/BatchPage.scala
Loading
Please register or sign in to comment