Skip to content
Snippets Groups Projects
  • Marcelo Vanzin's avatar
    c2467dad
    [SPARK-11140][CORE] Transfer files using network lib when using NettyRpcEnv. · c2467dad
    Marcelo Vanzin authored
    This change abstracts the code that serves jars / files to executors so that
    each RpcEnv can have its own implementation; the akka version uses the existing
    HTTP-based file serving mechanism, while the netty versions uses the new
    stream support added to the network lib, which makes file transfers benefit
    from the easier security configuration of the network library, and should also
    reduce overhead overall.
    
    The change includes a small fix to TransportChannelHandler so that it propagates
    user events to downstream handlers.
    
    Author: Marcelo Vanzin <vanzin@cloudera.com>
    
    Closes #9530 from vanzin/SPARK-11140.
    c2467dad
    History
    [SPARK-11140][CORE] Transfer files using network lib when using NettyRpcEnv.
    Marcelo Vanzin authored
    This change abstracts the code that serves jars / files to executors so that
    each RpcEnv can have its own implementation; the akka version uses the existing
    HTTP-based file serving mechanism, while the netty versions uses the new
    stream support added to the network lib, which makes file transfers benefit
    from the easier security configuration of the network library, and should also
    reduce overhead overall.
    
    The change includes a small fix to TransportChannelHandler so that it propagates
    user events to downstream handlers.
    
    Author: Marcelo Vanzin <vanzin@cloudera.com>
    
    Closes #9530 from vanzin/SPARK-11140.