-
- Downloads
[SPARK-21599][SQL] Collecting column statistics for datasource tables may fail...
[SPARK-21599][SQL] Collecting column statistics for datasource tables may fail with java.util.NoSuchElementException ## What changes were proposed in this pull request? In case of datasource tables (when they are stored in non-hive compatible way) , the schema information is recorded as table properties in hive meta-store. The alterTableStats method needs to get the schema information from table properties for data source tables before recording the column level statistics. Currently, we don't get the correct schema information and fail with java.util.NoSuchElement exception. ## How was this patch tested? A new test case is added in StatisticsSuite. Author: Dilip Biswal <dbiswal@us.ibm.com> Closes #18804 from dilipbiswal/datasource_stats.
Showing
- sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala 6 additions, 1 deletion...scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/StatisticsSuite.scala 75 additions, 1 deletion...est/scala/org/apache/spark/sql/hive/StatisticsSuite.scala
Loading
Please register or sign in to comment