-
- Downloads
[SPARK-18029][SQL] PruneFileSourcePartitions should not change the output of LogicalRelation
## What changes were proposed in this pull request? In `PruneFileSourcePartitions`, we will replace the `LogicalRelation` with a pruned one. However, this replacement may change the output of the `LogicalRelation` if it doesn't have `expectedOutputAttributes`. This PR fixes it. ## How was this patch tested? the new `PruneFileSourcePartitionsSuite` Author: Wenchen Fan <wenchen@databricks.com> Closes #15569 from cloud-fan/partition-bug.
Showing
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala 2 additions, 2 deletions...ala/org/apache/spark/sql/catalyst/catalog/interface.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PruneFileSourcePartitions.scala 3 additions, 1 deletion...sql/execution/datasources/PruneFileSourcePartitions.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveDataFrameSuite.scala 4 additions, 3 deletions.../scala/org/apache/spark/sql/hive/HiveDataFrameSuite.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveMetadataCacheSuite.scala 2 additions, 1 deletion...la/org/apache/spark/sql/hive/HiveMetadataCacheSuite.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/PruneFileSourcePartitionsSuite.scala 74 additions, 0 deletions...k/sql/hive/execution/PruneFileSourcePartitionsSuite.scala
Loading
Please register or sign in to comment