-
- Downloads
[SPARK-1402] Added 3 more compression schemes
JIRA issue: [SPARK-1402](https://issues.apache.org/jira/browse/SPARK-1402) This PR provides 3 more compression schemes for Spark SQL in-memory columnar storage: * `BooleanBitSet` * `IntDelta` * `LongDelta` Now there are 6 compression schemes in total, including the no-op `PassThrough` scheme. Also fixed a bug in PR #286: not all compression schemes are added as available schemes when accessing an in-memory column, and when a column is compressed with an unrecognised scheme, `ColumnAccessor` throws exception. Author: Cheng Lian <lian.cs.zju@gmail.com> Closes #330 from liancheng/moreCompressionSchemes and squashes the following commits: 1d037b8 [Cheng Lian] Fixed SPARK-1436: in-memory column byte buffer must be able to be accessed multiple times d7c0e8f [Cheng Lian] Added test suite for IntegralDelta (IntDelta & LongDelta) 3c1ad7a [Cheng Lian] Added test suite for BooleanBitSet, refactored other test suites 44fe4b2 [Cheng Lian] Refactored CompressionScheme, added 3 more compression schemes.
Showing
- sql/core/src/main/scala/org/apache/spark/sql/columnar/ColumnAccessor.scala 12 additions, 11 deletions.../scala/org/apache/spark/sql/columnar/ColumnAccessor.scala
- sql/core/src/main/scala/org/apache/spark/sql/columnar/ColumnStats.scala 6 additions, 0 deletions...ain/scala/org/apache/spark/sql/columnar/ColumnStats.scala
- sql/core/src/main/scala/org/apache/spark/sql/columnar/compression/CompressibleColumnBuilder.scala 3 additions, 3 deletions.../sql/columnar/compression/CompressibleColumnBuilder.scala
- sql/core/src/main/scala/org/apache/spark/sql/columnar/compression/CompressionScheme.scala 13 additions, 15 deletions...he/spark/sql/columnar/compression/CompressionScheme.scala
- sql/core/src/main/scala/org/apache/spark/sql/columnar/compression/compressionSchemes.scala 230 additions, 36 deletions...e/spark/sql/columnar/compression/compressionSchemes.scala
- sql/core/src/test/scala/org/apache/spark/sql/columnar/ColumnarQuerySuite.scala 8 additions, 0 deletions...la/org/apache/spark/sql/columnar/ColumnarQuerySuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/columnar/compression/BooleanBitSetSuite.scala 98 additions, 0 deletions...e/spark/sql/columnar/compression/BooleanBitSetSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/columnar/compression/DictionaryEncodingSuite.scala 65 additions, 57 deletions...rk/sql/columnar/compression/DictionaryEncodingSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/columnar/compression/IntegralDeltaSuite.scala 115 additions, 0 deletions...e/spark/sql/columnar/compression/IntegralDeltaSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/columnar/compression/RunLengthEncodingSuite.scala 32 additions, 55 deletions...ark/sql/columnar/compression/RunLengthEncodingSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/columnar/compression/TestCompressibleColumnBuilder.scala 4 additions, 2 deletions.../columnar/compression/TestCompressibleColumnBuilder.scala
Loading
Please register or sign in to comment