-
- Downloads
[SPARK-17992][SQL] Return all partitions from HiveShim when Hive throws a...
[SPARK-17992][SQL] Return all partitions from HiveShim when Hive throws a metastore exception when attempting to fetch partitions by filter (Link to Jira issue: https://issues.apache.org/jira/browse/SPARK-17992) ## What changes were proposed in this pull request? We recently added table partition pruning for partitioned Hive tables converted to using `TableFileCatalog`. When the Hive configuration option `hive.metastore.try.direct.sql` is set to `false`, Hive will throw an exception for unsupported filter expressions. For example, attempting to filter on an integer partition column will throw a `org.apache.hadoop.hive.metastore.api.MetaException`. I discovered this behavior because VideoAmp uses the CDH version of Hive with a Postgresql metastore DB. In this configuration, CDH sets `hive.metastore.try.direct.sql` to `false` by default, and queries that filter on a non-string partition column will fail. Rather than throw an exception in query planning, this patch catches this exception, logs a warning and returns all table partitions instead. Clients of this method are already expected to handle the possibility that the filters will not be honored. ## How was this patch tested? A unit test was added. Author: Michael Allman <michael@videoamp.com> Closes #15673 from mallman/spark-17992-catch_hive_partition_filter_exception.
Showing
- sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala 23 additions, 8 deletions...ain/scala/org/apache/spark/sql/hive/client/HiveShim.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HiveClientBuilder.scala 56 additions, 0 deletions.../org/apache/spark/sql/hive/client/HiveClientBuilder.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HiveClientSuite.scala 61 additions, 0 deletions...la/org/apache/spark/sql/hive/client/HiveClientSuite.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/client/VersionsSuite.scala 20 additions, 57 deletions...cala/org/apache/spark/sql/hive/client/VersionsSuite.scala
Loading
Please register or sign in to comment