Skip to content
Snippets Groups Projects
Commit e597ea34 authored by Matei Zaharia's avatar Matei Zaharia
Browse files

Merge pull request #10 from kayousterhout/results_through-bm

Send Task results through the block manager when larger than Akka frame size (fixes SPARK-669).

This change requires adding an extra failure mode: tasks can complete
successfully, but the result gets lost or flushed from the block manager
before it's been fetched.

This change also moves the deserialization of tasks into a separate thread, so it's no longer part of the DAG scheduler's tight loop. This should improve scheduler throughput, particularly when tasks are sending back large results.

Thanks Josh for writing the original version of this patch!

This is duplicated from the mesos/spark repo: https://github.com/mesos/spark/pull/835
parents 714fdabd 0dcad2ed
No related branches found
No related tags found
No related merge requests found
Showing
with 496 additions and 167 deletions
Loading
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment