Skip to content
  • Marcelo Vanzin's avatar
    1c31ebd1
    [SPARK-6578] [core] Fix thread-safety issue in outbound path of network library. · 1c31ebd1
    Marcelo Vanzin authored
    
    
    While the inbound path of a netty pipeline is thread-safe, the outbound
    path is not. That means that multiple threads can compete to write messages
    to the next stage of the pipeline.
    
    The network library sometimes breaks a single RPC message into multiple
    buffers internally to avoid copying data (see MessageEncoder). This can
    result in the following scenario (where "FxBy" means "frame x, buffer y"):
    
                   T1         F1B1            F1B2
                                \               \
                                 \               \
                   socket        F1B1   F2B1    F1B2  F2B2
                                         /             /
                                        /             /
                   T2                  F2B1         F2B2
    
    And the frames now cannot be rebuilt on the receiving side because the
    different messages have been mixed up on the wire.
    
    The fix wraps these multi-buffer messages into a `FileRegion` object
    so that these messages are written "atomically" to the next pipeline handler.
    
    Author: Marcelo Vanzin <vanzin@cloudera.com>
    
    Closes #5234 from vanzin/SPARK-6578 and squashes the following commits:
    
    16b2d70 [Marcelo Vanzin] Forgot to update a type.
    c9c2e4e [Marcelo Vanzin] Review comments: simplify some code.
    9c888ac [Marcelo Vanzin] Small style nits.
    8474bab [Marcelo Vanzin] Fix multiple calls to MessageWithHeader.transferTo().
    e26509f [Marcelo Vanzin] Merge branch 'master' into SPARK-6578
    c503f6c [Marcelo Vanzin] Implement a custom FileRegion instead of using locks.
    84aa7ce [Marcelo Vanzin] Rename handler to the correct name.
    432f3bd [Marcelo Vanzin] Remove unneeded method.
    8d70e60 [Marcelo Vanzin] Fix thread-safety issue in outbound path of network library.
    
    (cherry picked from commit f084c5de)
    Signed-off-by: default avatarReynold Xin <rxin@databricks.com>
    1c31ebd1
    [SPARK-6578] [core] Fix thread-safety issue in outbound path of network library.
    Marcelo Vanzin authored
    
    
    While the inbound path of a netty pipeline is thread-safe, the outbound
    path is not. That means that multiple threads can compete to write messages
    to the next stage of the pipeline.
    
    The network library sometimes breaks a single RPC message into multiple
    buffers internally to avoid copying data (see MessageEncoder). This can
    result in the following scenario (where "FxBy" means "frame x, buffer y"):
    
                   T1         F1B1            F1B2
                                \               \
                                 \               \
                   socket        F1B1   F2B1    F1B2  F2B2
                                         /             /
                                        /             /
                   T2                  F2B1         F2B2
    
    And the frames now cannot be rebuilt on the receiving side because the
    different messages have been mixed up on the wire.
    
    The fix wraps these multi-buffer messages into a `FileRegion` object
    so that these messages are written "atomically" to the next pipeline handler.
    
    Author: Marcelo Vanzin <vanzin@cloudera.com>
    
    Closes #5234 from vanzin/SPARK-6578 and squashes the following commits:
    
    16b2d70 [Marcelo Vanzin] Forgot to update a type.
    c9c2e4e [Marcelo Vanzin] Review comments: simplify some code.
    9c888ac [Marcelo Vanzin] Small style nits.
    8474bab [Marcelo Vanzin] Fix multiple calls to MessageWithHeader.transferTo().
    e26509f [Marcelo Vanzin] Merge branch 'master' into SPARK-6578
    c503f6c [Marcelo Vanzin] Implement a custom FileRegion instead of using locks.
    84aa7ce [Marcelo Vanzin] Rename handler to the correct name.
    432f3bd [Marcelo Vanzin] Remove unneeded method.
    8d70e60 [Marcelo Vanzin] Fix thread-safety issue in outbound path of network library.
    
    (cherry picked from commit f084c5de)
    Signed-off-by: default avatarReynold Xin <rxin@databricks.com>
Loading