-
- Downloads
[SPARK-21276][CORE] Update lz4-java to the latest (v1.4.0)
## What changes were proposed in this pull request? This pr updated `lz4-java` to the latest (v1.4.0) and removed custom `LZ4BlockInputStream`. We currently use custom `LZ4BlockInputStream` to read concatenated byte stream in shuffle. But, this functionality has been implemented in the latest lz4-java (https://github.com/lz4/lz4-java/pull/105). So, we might update the latest to remove the custom `LZ4BlockInputStream`. Major diffs between the latest release and v1.3.0 in the master are as follows (https://github.com/lz4/lz4-java/compare/62f7547abb0819d1ca1e669645ee1a9d26cd60b0...6d4693f56253fcddfad7b441bb8d917b182efa2d); - fixed NPE in XXHashFactory similarly - Don't place resources in default package to support shading - Fixes ByteBuffer methods failing to apply arrayOffset() for array-backed - Try to load lz4-java from java.library.path, then fallback to bundled - Add ppc64le binary - Add s390x JNI binding - Add basic LZ4 Frame v1.5.0 support - enable aarch64 support for lz4-java - Allow unsafeInstance() for ppc64le archiecture - Add unsafeInstance support for AArch64 - Support 64-bit JNI build on Solaris - Avoid over-allocating a buffer - Allow EndMark to be incompressible for LZ4FrameInputStream. - Concat byte stream ## How was this patch tested? Existing tests. Author: Takeshi Yamamuro <yamamuro@apache.org> Closes #18883 from maropu/SPARK-21276.
Showing
- core/pom.xml 2 additions, 2 deletionscore/pom.xml
- core/src/main/java/org/apache/spark/io/LZ4BlockInputStream.java 0 additions, 260 deletions...rc/main/java/org/apache/spark/io/LZ4BlockInputStream.java
- core/src/main/scala/org/apache/spark/io/CompressionCodec.scala 5 additions, 2 deletions...src/main/scala/org/apache/spark/io/CompressionCodec.scala
- dev/deps/spark-deps-hadoop-2.6 1 addition, 1 deletiondev/deps/spark-deps-hadoop-2.6
- dev/deps/spark-deps-hadoop-2.7 1 addition, 1 deletiondev/deps/spark-deps-hadoop-2.7
- external/kafka-0-10-assembly/pom.xml 2 additions, 2 deletionsexternal/kafka-0-10-assembly/pom.xml
- external/kafka-0-8-assembly/pom.xml 2 additions, 2 deletionsexternal/kafka-0-8-assembly/pom.xml
- pom.xml 3 additions, 3 deletionspom.xml
- project/MimaExcludes.scala 4 additions, 1 deletionproject/MimaExcludes.scala
Loading
Please register or sign in to comment