-
- Downloads
[SPARK-11246] [SQL] Table cache for Parquet broken in 1.5
The root cause is that when spark.sql.hive.convertMetastoreParquet=true by default, the cached InMemoryRelation of the ParquetRelation can not be looked up from the cachedData of CacheManager because the key comparison fails even though it is the same LogicalPlan representing the Subquery that wraps the ParquetRelation. The solution in this PR is overriding the LogicalPlan.sameResult function in Subquery case class to eliminate subquery node first before directly comparing the child (ParquetRelation), which will find the key to the cached InMemoryRelation. Author: xin Wu <xinwu@us.ibm.com> Closes #9326 from xwu0226/spark-11246-commit.
Showing
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/LogicalRelation.scala 5 additions, 0 deletions...che/spark/sql/execution/datasources/LogicalRelation.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/CachedTableSuite.scala 11 additions, 0 deletions...st/scala/org/apache/spark/sql/hive/CachedTableSuite.scala
Please register or sign in to comment