-
- Downloads
[SPARK-17980][SQL] Fix refreshByPath for converted Hive tables
## What changes were proposed in this pull request? There was a bug introduced in https://github.com/apache/spark/pull/14690 which broke refreshByPath with converted hive tables (though, it turns out it was very difficult to refresh converted hive tables anyways, since you had to specify the exact path of one of the partitions). This changes refreshByPath to invalidate by prefix instead of exact match, and fixes the issue. cc sameeragarwal for refreshByPath changes mallman ## How was this patch tested? Extended unit test. Author: Eric Liang <ekl@databricks.com> Closes #15521 from ericl/fix-caching.
Showing
- sql/core/src/main/scala/org/apache/spark/sql/catalog/Catalog.scala 2 additions, 1 deletion...src/main/scala/org/apache/spark/sql/catalog/Catalog.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/CacheManager.scala 3 additions, 2 deletions...n/scala/org/apache/spark/sql/execution/CacheManager.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/TableFileCatalog.scala 14 additions, 4 deletions...he/spark/sql/execution/datasources/TableFileCatalog.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala 1 addition, 1 deletion...cala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveMetadataCacheSuite.scala 19 additions, 2 deletions...la/org/apache/spark/sql/hive/HiveMetadataCacheSuite.scala
Loading
Please register or sign in to comment