-
- Downloads
[SPARK-19326] Speculated task attempts do not get launched in few scenarios
## What changes were proposed in this pull request? Add a new listener event when a speculative task is created and notify it to ExecutorAllocationManager for requesting more executor. ## How was this patch tested? - Added Unittests. - For the test snippet in the jira: val n = 100 val someRDD = sc.parallelize(1 to n, n) someRDD.mapPartitionsWithIndex( (index: Int, it: Iterator[Int]) => { if (index == 1) { Thread.sleep(Long.MaxValue) // fake long running task(s) } it.toList.map(x => index + ", " + x).iterator }).collect With this code change, spark indicates 101 jobs are running (99 succeeded, 2 running and 1 is speculative job) Author: Jane Wang <janewang@fb.com> Closes #18492 from janewangfb/speculated_task_not_launched.
Showing
- core/src/main/java/org/apache/spark/SparkFirehoseListener.java 5 additions, 0 deletions...src/main/java/org/apache/spark/SparkFirehoseListener.java
- core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala 52 additions, 9 deletions...in/scala/org/apache/spark/ExecutorAllocationManager.scala
- core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala 14 additions, 0 deletions.../main/scala/org/apache/spark/scheduler/DAGScheduler.scala
- core/src/main/scala/org/apache/spark/scheduler/DAGSchedulerEvent.scala 4 additions, 0 deletions.../scala/org/apache/spark/scheduler/DAGSchedulerEvent.scala
- core/src/main/scala/org/apache/spark/scheduler/SparkListener.scala 11 additions, 0 deletions...main/scala/org/apache/spark/scheduler/SparkListener.scala
- core/src/main/scala/org/apache/spark/scheduler/SparkListenerBus.scala 2 additions, 0 deletions...n/scala/org/apache/spark/scheduler/SparkListenerBus.scala
- core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala 1 addition, 0 deletions...ain/scala/org/apache/spark/scheduler/TaskSetManager.scala
- core/src/test/scala/org/apache/spark/ExecutorAllocationManagerSuite.scala 46 additions, 2 deletions...ala/org/apache/spark/ExecutorAllocationManagerSuite.scala
- core/src/test/scala/org/apache/spark/scheduler/TaskSetManagerSuite.scala 9 additions, 0 deletions...cala/org/apache/spark/scheduler/TaskSetManagerSuite.scala
Loading
Please register or sign in to comment