-
- Downloads
[SPARK-1292] In-memory columnar representation for Spark SQL
This PR is rebased from the Catalyst repository, and contains the first version of in-memory columnar representation for Spark SQL. Compression support is not included yet and will be added later in a separate PR. Author: Cheng Lian <lian@databricks.com> Author: Cheng Lian <lian.cs.zju@gmail.com> Closes #205 from liancheng/memColumnarSupport and squashes the following commits: 99dba41 [Cheng Lian] Restricted new objects/classes to `private[sql]' 0892ad8 [Cheng Lian] Addressed ScalaStyle issues af1ad5e [Cheng Lian] Fixed some minor issues introduced during rebasing 0dbf2fb [Cheng Lian] Make necessary renaming due to rebase a162d4d [Cheng Lian] Removed the unnecessary InMemoryColumnarRelation class 9bcae4b [Cheng Lian] Added Apache license 220ee1e [Cheng Lian] Added table scan operator for in-memory columnar support. c701c7a [Cheng Lian] Using SparkSqlSerializer for generic object SerDe causes error, made a workaround ed8608e [Cheng Lian] Added implicit conversion from DataType to ColumnType b8a645a [Cheng Lian] Replaced KryoSerializer with an updated SparkSqlSerializer b6c0a49 [Cheng Lian] Minor test suite refactoring 214be73 [Cheng Lian] Refactored BINARY and GENERIC to reduce duplicate code da2f4d5 [Cheng Lian] Added Apache license dbf7a38 [Cheng Lian] Added ColumnAccessor and test suite, refactored ColumnBuilder c01a177 [Cheng Lian] Added column builder classes and test suite f18ddc6 [Cheng Lian] Added ColumnTypes and test suite 2d09066 [Cheng Lian] Added KryoSerializer 34f3c19 [Cheng Lian] Added TypeTag field to all NativeTypes acc5c48 [Cheng Lian] Added Hive test files to .gitignore
Showing
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/types/dataTypes.scala 13 additions, 1 deletion...scala/org/apache/spark/sql/catalyst/types/dataTypes.scala
- sql/core/src/main/scala/org/apache/spark/sql/columnar/ColumnAccessor.scala 175 additions, 0 deletions.../scala/org/apache/spark/sql/columnar/ColumnAccessor.scala
- sql/core/src/main/scala/org/apache/spark/sql/columnar/ColumnBuilder.scala 187 additions, 0 deletions...n/scala/org/apache/spark/sql/columnar/ColumnBuilder.scala
- sql/core/src/main/scala/org/apache/spark/sql/columnar/ColumnType.scala 198 additions, 0 deletions...main/scala/org/apache/spark/sql/columnar/ColumnType.scala
- sql/core/src/main/scala/org/apache/spark/sql/columnar/NullableColumnAccessor.scala 57 additions, 0 deletions...rg/apache/spark/sql/columnar/NullableColumnAccessor.scala
- sql/core/src/main/scala/org/apache/spark/sql/columnar/NullableColumnBuilder.scala 83 additions, 0 deletions...org/apache/spark/sql/columnar/NullableColumnBuilder.scala
- sql/core/src/main/scala/org/apache/spark/sql/columnar/inMemoryColumnarOperators.scala 80 additions, 0 deletions...apache/spark/sql/columnar/inMemoryColumnarOperators.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/Exchange.scala 0 additions, 33 deletions.../main/scala/org/apache/spark/sql/execution/Exchange.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlSerializer.scala 73 additions, 0 deletions...a/org/apache/spark/sql/execution/SparkSqlSerializer.scala
- sql/core/src/test/scala/org/apache/spark/sql/QueryTest.scala 1 addition, 1 deletionsql/core/src/test/scala/org/apache/spark/sql/QueryTest.scala
- sql/core/src/test/scala/org/apache/spark/sql/columnar/ColumnTypeSuite.scala 204 additions, 0 deletions...scala/org/apache/spark/sql/columnar/ColumnTypeSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/columnar/ColumnarQuerySuite.scala 34 additions, 0 deletions...la/org/apache/spark/sql/columnar/ColumnarQuerySuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/columnar/ColumnarTestData.scala 55 additions, 0 deletions...cala/org/apache/spark/sql/columnar/ColumnarTestData.scala
- sql/core/src/test/scala/org/apache/spark/sql/columnar/NullableColumnAccessorSuite.scala 61 additions, 0 deletions...ache/spark/sql/columnar/NullableColumnAccessorSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/columnar/NullableColumnBuilderSuite.scala 94 additions, 0 deletions...pache/spark/sql/columnar/NullableColumnBuilderSuite.scala
Loading
Please register or sign in to comment