-
- Downloads
[BUGFIX] In-memory columnar storage bug fixes
Fixed several bugs of in-memory columnar storage to make `HiveInMemoryCompatibilitySuite` pass. @rxin @marmbrus It is reasonable to include `HiveInMemoryCompatibilitySuite` in this PR, but I didn't, since it significantly increases test execution time. What do you think? **UPDATE** `HiveCompatibilitySuite` has been made to cache tables in memory. `HiveInMemoryCompatibilitySuite` was removed. Author: Cheng Lian <lian.cs.zju@gmail.com> Author: Michael Armbrust <michael@databricks.com> Closes #374 from liancheng/inMemBugFix and squashes the following commits: 6ad6d9b [Cheng Lian] Merged HiveCompatibilitySuite and HiveInMemoryCompatibilitySuite 5bdbfe7 [Cheng Lian] Revert 882c538 & 8426ddc, which introduced regression 882c538 [Cheng Lian] Remove attributes field from InMemoryColumnarTableScan 32cc9ce [Cheng Lian] Code style cleanup 99382bf [Cheng Lian] Enable compression by default 4390bcc [Cheng Lian] Report error for any Throwable in HiveComparisonTest d1df4fd [Michael Armbrust] Remove test tables that might always get created anyway? ab9e807 [Michael Armbrust] Fix the logged console version of failed test cases to use the new syntax. 1965123 [Michael Armbrust] Don't use coalesce for gathering all data to a single partition, as it does not work correctly with mutable rows. e36cdd0 [Michael Armbrust] Spelling. 2d0e168 [Michael Armbrust] Run Hive tests in-memory too. 6360723 [Cheng Lian] Made PreInsertionCasts support SparkLogicalPlan and InMemoryColumnarTableScan c9b0f6f [Cheng Lian] Let InsertIntoTable support InMemoryColumnarTableScan 9c8fc40 [Cheng Lian] Disable compression by default e619995 [Cheng Lian] Bug fix: incorrect byte order in CompressionScheme.columnHeaderSize 8426ddc [Cheng Lian] Bug fix: InMemoryColumnarTableScan should cache columns specified by the attributes argument 036cd09 [Cheng Lian] Clean up unused imports 44591a5 [Cheng Lian] Bug fix: NullableColumnAccessor.hasNext must take nulls into account 052bf41 [Cheng Lian] Bug fix: should only gather compressibility info for non-null values 95b3301 [Cheng Lian] Fixed bugs in IntegralDelta
Showing
- sql/core/src/main/scala/org/apache/spark/sql/SchemaRDD.scala 1 addition, 1 deletionsql/core/src/main/scala/org/apache/spark/sql/SchemaRDD.scala
- sql/core/src/main/scala/org/apache/spark/sql/columnar/NullableColumnAccessor.scala 2 additions, 0 deletions...rg/apache/spark/sql/columnar/NullableColumnAccessor.scala
- sql/core/src/main/scala/org/apache/spark/sql/columnar/compression/CompressibleColumnBuilder.scala 3 additions, 1 deletion.../sql/columnar/compression/CompressibleColumnBuilder.scala
- sql/core/src/main/scala/org/apache/spark/sql/columnar/compression/CompressionScheme.scala 2 additions, 2 deletions...he/spark/sql/columnar/compression/CompressionScheme.scala
- sql/core/src/main/scala/org/apache/spark/sql/columnar/compression/compressionSchemes.scala 8 additions, 12 deletions...e/spark/sql/columnar/compression/compressionSchemes.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/Exchange.scala 8 additions, 1 deletion.../main/scala/org/apache/spark/sql/execution/Exchange.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/SparkPlan.scala 2 additions, 2 deletions...main/scala/org/apache/spark/sql/execution/SparkPlan.scala
- sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala 2 additions, 3 deletions...rc/test/scala/org/apache/spark/sql/CachedTableSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/columnar/InMemoryColumnarQuerySuite.scala 11 additions, 1 deletion...pache/spark/sql/columnar/InMemoryColumnarQuerySuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/columnar/NullableColumnAccessorSuite.scala 4 additions, 0 deletions...ache/spark/sql/columnar/NullableColumnAccessorSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/columnar/compression/IntegralDeltaSuite.scala 11 additions, 4 deletions...e/spark/sql/columnar/compression/IntegralDeltaSuite.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala 26 additions, 16 deletions...cala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveStrategies.scala 4 additions, 0 deletions...main/scala/org/apache/spark/sql/hive/HiveStrategies.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/TestHive.scala 4 additions, 6 deletions...e/src/main/scala/org/apache/spark/sql/hive/TestHive.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/hiveUdfs.scala 8 additions, 8 deletions...e/src/main/scala/org/apache/spark/sql/hive/hiveUdfs.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveComparisonTest.scala 2 additions, 8 deletions.../apache/spark/sql/hive/execution/HiveComparisonTest.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveCompatibilitySuite.scala 11 additions, 1 deletion...che/spark/sql/hive/execution/HiveCompatibilitySuite.scala
Loading
Please register or sign in to comment