-
- Downloads
Merge pull request #10 from kayousterhout/results_through-bm
Send Task results through the block manager when larger than Akka frame size (fixes SPARK-669). This change requires adding an extra failure mode: tasks can complete successfully, but the result gets lost or flushed from the block manager before it's been fetched. This change also moves the deserialization of tasks into a separate thread, so it's no longer part of the DAG scheduler's tight loop. This should improve scheduler throughput, particularly when tasks are sending back large results. Thanks Josh for writing the original version of this patch! This is duplicated from the mesos/spark repo: https://github.com/mesos/spark/pull/835
No related branches found
No related tags found
Showing
- core/src/main/scala/org/apache/spark/SparkContext.scala 1 addition, 1 deletioncore/src/main/scala/org/apache/spark/SparkContext.scala
- core/src/main/scala/org/apache/spark/TaskEndReason.scala 6 additions, 2 deletionscore/src/main/scala/org/apache/spark/TaskEndReason.scala
- core/src/main/scala/org/apache/spark/executor/Executor.scala 17 additions, 9 deletionscore/src/main/scala/org/apache/spark/executor/Executor.scala
- core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala 4 additions, 1 deletion.../main/scala/org/apache/spark/scheduler/DAGScheduler.scala
- core/src/main/scala/org/apache/spark/scheduler/Pool.scala 3 additions, 3 deletionscore/src/main/scala/org/apache/spark/scheduler/Pool.scala
- core/src/main/scala/org/apache/spark/scheduler/Schedulable.scala 1 addition, 3 deletions...c/main/scala/org/apache/spark/scheduler/Schedulable.scala
- core/src/main/scala/org/apache/spark/scheduler/TaskResult.scala 10 additions, 5 deletions...rc/main/scala/org/apache/spark/scheduler/TaskResult.scala
- core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala 0 additions, 2 deletions...ain/scala/org/apache/spark/scheduler/TaskSetManager.scala
- core/src/main/scala/org/apache/spark/scheduler/cluster/ClusterScheduler.scala 42 additions, 13 deletions...org/apache/spark/scheduler/cluster/ClusterScheduler.scala
- core/src/main/scala/org/apache/spark/scheduler/cluster/ClusterTaskSetManager.scala 74 additions, 88 deletions...pache/spark/scheduler/cluster/ClusterTaskSetManager.scala
- core/src/main/scala/org/apache/spark/scheduler/cluster/TaskResultGetter.scala 124 additions, 0 deletions...org/apache/spark/scheduler/cluster/TaskResultGetter.scala
- core/src/main/scala/org/apache/spark/scheduler/local/LocalScheduler.scala 3 additions, 2 deletions...ala/org/apache/spark/scheduler/local/LocalScheduler.scala
- core/src/main/scala/org/apache/spark/scheduler/local/LocalTaskSetManager.scala 13 additions, 8 deletions...rg/apache/spark/scheduler/local/LocalTaskSetManager.scala
- core/src/main/scala/org/apache/spark/storage/BlockManager.scala 26 additions, 1 deletion...rc/main/scala/org/apache/spark/storage/BlockManager.scala
- core/src/test/scala/org/apache/spark/DistributedSuite.scala 0 additions, 13 deletionscore/src/test/scala/org/apache/spark/DistributedSuite.scala
- core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala 0 additions, 2 deletions.../scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
- core/src/test/scala/org/apache/spark/scheduler/cluster/ClusterSchedulerSuite.scala 8 additions, 7 deletions...pache/spark/scheduler/cluster/ClusterSchedulerSuite.scala
- core/src/test/scala/org/apache/spark/scheduler/cluster/ClusterTaskSetManagerSuite.scala 51 additions, 7 deletions.../spark/scheduler/cluster/ClusterTaskSetManagerSuite.scala
- core/src/test/scala/org/apache/spark/scheduler/cluster/TaskResultGetterSuite.scala 113 additions, 0 deletions...pache/spark/scheduler/cluster/TaskResultGetterSuite.scala
Loading
Please register or sign in to comment