-
- Downloads
[SPARK-4182][SQL] Fixes ColumnStats classes for boolean, binary and complex data types
`NoopColumnStats` was once used for binary, boolean and complex data types. This `ColumnStats` doesn't return properly shaped column statistics and causes caching failure if a table contains columns of the aforementioned types. This PR adds `BooleanColumnStats`, `BinaryColumnStats` and `GenericColumnStats`, used for boolean, binary and all complex data types respectively. In addition, `NoopColumnStats` returns properly shaped column statistics containing null count and row count, but this class is now used for testing purpose only. Author: Cheng Lian <lian@databricks.com> Closes #3059 from liancheng/spark-4182 and squashes the following commits: b398cfd [Cheng Lian] Fixes failed test case fb3ee85 [Cheng Lian] Fixes SPARK-4182
Showing
- sql/core/src/main/scala/org/apache/spark/sql/columnar/ColumnBuilder.scala 6 additions, 4 deletions...n/scala/org/apache/spark/sql/columnar/ColumnBuilder.scala
- sql/core/src/main/scala/org/apache/spark/sql/columnar/ColumnStats.scala 43 additions, 2 deletions...ain/scala/org/apache/spark/sql/columnar/ColumnStats.scala
- sql/core/src/main/scala/org/apache/spark/sql/columnar/InMemoryColumnarTableScan.scala 3 additions, 0 deletions...apache/spark/sql/columnar/InMemoryColumnarTableScan.scala
- sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala 4 additions, 3 deletions...e/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/TestData.scala 8 additions, 0 deletionssql/core/src/test/scala/org/apache/spark/sql/TestData.scala
- sql/core/src/test/scala/org/apache/spark/sql/columnar/InMemoryColumnarQuerySuite.scala 18 additions, 10 deletions...pache/spark/sql/columnar/InMemoryColumnarQuerySuite.scala
Loading
Please register or sign in to comment