-
- Downloads
[SPARK-16929] Improve performance when check speculatable tasks.
## What changes were proposed in this pull request? 1. Use a MedianHeap to record durations of successful tasks. When check speculatable tasks, we can get the median duration with O(1) time complexity. 2. `checkSpeculatableTasks` will synchronize `TaskSchedulerImpl`. If `checkSpeculatableTasks` doesn't finish with 100ms, then the possibility exists for that thread to release and then immediately re-acquire the lock. Change `scheduleAtFixedRate` to be `scheduleWithFixedDelay` when call method of `checkSpeculatableTasks`. ## How was this patch tested? Added MedianHeapSuite. Author: jinxing <jinxing6042@126.com> Closes #16867 from jinxing64/SPARK-16929.
Showing
- core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala 1 addition, 1 deletion.../scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
- core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala 14 additions, 5 deletions...ain/scala/org/apache/spark/scheduler/TaskSetManager.scala
- core/src/main/scala/org/apache/spark/util/collection/MedianHeap.scala 93 additions, 0 deletions...n/scala/org/apache/spark/util/collection/MedianHeap.scala
- core/src/test/scala/org/apache/spark/scheduler/TaskSetManagerSuite.scala 2 additions, 0 deletions...cala/org/apache/spark/scheduler/TaskSetManagerSuite.scala
- core/src/test/scala/org/apache/spark/util/collection/MedianHeapSuite.scala 66 additions, 0 deletions...la/org/apache/spark/util/collection/MedianHeapSuite.scala
Loading
Please register or sign in to comment