-
- Downloads
[SPARK-15915][SQL] Logical plans should use canonicalized plan when override sameResult.
## What changes were proposed in this pull request? `DataFrame` with plan overriding `sameResult` but not using canonicalized plan to compare can't cacheTable. The example is like: ``` val localRelation = Seq(1, 2, 3).toDF() localRelation.createOrReplaceTempView("localRelation") spark.catalog.cacheTable("localRelation") assert( localRelation.queryExecution.withCachedData.collect { case i: InMemoryRelation => i }.size == 1) ``` and this will fail as: ``` ArrayBuffer() had size 0 instead of expected size 1 ``` The reason is that when do `spark.catalog.cacheTable("localRelation")`, `CacheManager` tries to cache for the plan wrapped by `SubqueryAlias` but when planning for the DataFrame `localRelation`, `CacheManager` tries to find cached table for the not-wrapped plan because the plan for DataFrame `localRelation` is not wrapped. Some plans like `LocalRelation`, `LogicalRDD`, etc. override `sameResult` method, but not use canonicalized plan to compare so the `CacheManager` can't detect the plans are the same. This pr modifies them to use canonicalized plan when override `sameResult` method. ## How was this patch tested? Added a test to check if DataFrame with plan overriding sameResult but not using canonicalized plan to compare can cacheTable. Author: Takuya UESHIN <ueshin@happy-camper.st> Closes #13638 from ueshin/issues/SPARK-15915.
Showing
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/LocalRelation.scala 6 additions, 4 deletions...ache/spark/sql/catalyst/plans/logical/LocalRelation.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/ExistingRDD.scala 5 additions, 3 deletions...in/scala/org/apache/spark/sql/execution/ExistingRDD.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/LogicalRelation.scala 5 additions, 3 deletions...che/spark/sql/execution/datasources/LogicalRelation.scala
- sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala 11 additions, 0 deletions...rc/test/scala/org/apache/spark/sql/CachedTableSuite.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/MetastoreRelation.scala 1 addition, 1 deletion...n/scala/org/apache/spark/sql/hive/MetastoreRelation.scala
Please register or sign in to comment