Skip to content
Snippets Groups Projects
  • Takeshi Yamamuro's avatar
    b78cf13b
    [SPARK-21276][CORE] Update lz4-java to the latest (v1.4.0) · b78cf13b
    Takeshi Yamamuro authored
    ## What changes were proposed in this pull request?
    This pr updated `lz4-java` to the latest (v1.4.0) and removed custom `LZ4BlockInputStream`. We currently use custom `LZ4BlockInputStream` to read concatenated byte stream in shuffle. But, this functionality has been implemented in the latest lz4-java (https://github.com/lz4/lz4-java/pull/105). So, we might update the latest to remove the custom `LZ4BlockInputStream`.
    
    Major diffs between the latest release and v1.3.0 in the master are as follows (https://github.com/lz4/lz4-java/compare/62f7547abb0819d1ca1e669645ee1a9d26cd60b0...6d4693f56253fcddfad7b441bb8d917b182efa2d);
    - fixed NPE in XXHashFactory similarly
    - Don't place resources in default package to support shading
    - Fixes ByteBuffer methods failing to apply arrayOffset() for array-backed
    - Try to load lz4-java from java.library.path, then fallback to bundled
    - Add ppc64le binary
    - Add s390x JNI binding
    - Add basic LZ4 Frame v1.5.0 support
    - enable aarch64 support for lz4-java
    - Allow unsafeInstance() for ppc64le archiecture
    - Add unsafeInstance support for AArch64
    - Support 64-bit JNI build on Solaris
    - Avoid over-allocating a buffer
    - Allow EndMark to be incompressible for LZ4FrameInputStream.
    - Concat byte stream
    
    ## How was this patch tested?
    Existing tests.
    
    Author: Takeshi Yamamuro <yamamuro@apache.org>
    
    Closes #18883 from maropu/SPARK-21276.
    b78cf13b
    History
    [SPARK-21276][CORE] Update lz4-java to the latest (v1.4.0)
    Takeshi Yamamuro authored
    ## What changes were proposed in this pull request?
    This pr updated `lz4-java` to the latest (v1.4.0) and removed custom `LZ4BlockInputStream`. We currently use custom `LZ4BlockInputStream` to read concatenated byte stream in shuffle. But, this functionality has been implemented in the latest lz4-java (https://github.com/lz4/lz4-java/pull/105). So, we might update the latest to remove the custom `LZ4BlockInputStream`.
    
    Major diffs between the latest release and v1.3.0 in the master are as follows (https://github.com/lz4/lz4-java/compare/62f7547abb0819d1ca1e669645ee1a9d26cd60b0...6d4693f56253fcddfad7b441bb8d917b182efa2d);
    - fixed NPE in XXHashFactory similarly
    - Don't place resources in default package to support shading
    - Fixes ByteBuffer methods failing to apply arrayOffset() for array-backed
    - Try to load lz4-java from java.library.path, then fallback to bundled
    - Add ppc64le binary
    - Add s390x JNI binding
    - Add basic LZ4 Frame v1.5.0 support
    - enable aarch64 support for lz4-java
    - Allow unsafeInstance() for ppc64le archiecture
    - Add unsafeInstance support for AArch64
    - Support 64-bit JNI build on Solaris
    - Avoid over-allocating a buffer
    - Allow EndMark to be incompressible for LZ4FrameInputStream.
    - Concat byte stream
    
    ## How was this patch tested?
    Existing tests.
    
    Author: Takeshi Yamamuro <yamamuro@apache.org>
    
    Closes #18883 from maropu/SPARK-21276.
pom.xml 16.26 KiB