Skip to content
Snippets Groups Projects
Commit c2467dad authored by Marcelo Vanzin's avatar Marcelo Vanzin
Browse files

[SPARK-11140][CORE] Transfer files using network lib when using NettyRpcEnv.

This change abstracts the code that serves jars / files to executors so that
each RpcEnv can have its own implementation; the akka version uses the existing
HTTP-based file serving mechanism, while the netty versions uses the new
stream support added to the network lib, which makes file transfers benefit
from the easier security configuration of the network library, and should also
reduce overhead overall.

The change includes a small fix to TransportChannelHandler so that it propagates
user events to downstream handlers.

Author: Marcelo Vanzin <vanzin@cloudera.com>

Closes #9530 from vanzin/SPARK-11140.
parent 7cfa4c6b
No related branches found
No related tags found
No related merge requests found
Showing
with 356 additions and 47 deletions
Loading
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment