-
- Downloads
[SPARK-14459][SQL] Detect relation partitioning and adjust the logical plan
## What changes were proposed in this pull request? This detects a relation's partitioning and adds checks to the analyzer. If an InsertIntoTable node has no partitioning, it is replaced by the relation's partition scheme and input columns are correctly adjusted, placing the partition columns at the end in partition order. If an InsertIntoTable node has partitioning, it is checked against the table's reported partitions. These changes required adding a PartitionedRelation trait to the catalog interface because Hive's MetastoreRelation doesn't extend CatalogRelation. This commit also includes a fix to InsertIntoTable's resolved logic, which now detects that all expected columns are present, including dynamic partition columns. Previously, the number of expected columns was not checked and resolved was true if there were missing columns. ## How was this patch tested? This adds new tests to the InsertIntoTableSuite that are fixed by this PR. Author: Ryan Blue <blue@apache.org> Closes #12239 from rdblue/SPARK-14459-detect-hive-partitioning.
Showing
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala 38 additions, 3 deletions...ala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala 16 additions, 3 deletions...rk/sql/catalyst/plans/logical/basicLogicalOperators.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala 9 additions, 1 deletion...apache/spark/sql/hive/execution/InsertIntoHiveTable.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/InsertIntoHiveTableSuite.scala 79 additions, 4 deletions.../org/apache/spark/sql/hive/InsertIntoHiveTableSuite.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveQuerySuite.scala 1 addition, 1 deletion.../org/apache/spark/sql/hive/execution/HiveQuerySuite.scala
Loading
Please register or sign in to comment