-
- Downloads
[SPARK-19359][SQL] clear useless path after rename a partition with upper-case...
[SPARK-19359][SQL] clear useless path after rename a partition with upper-case by HiveExternalCatalog ## What changes were proposed in this pull request? Hive metastore is not case preserving and keep partition columns with lower case names. If SparkSQL create a table with upper-case partion name use HiveExternalCatalog, when we rename partition, it first call the HiveClient to renamePartition, which will create a new lower case partition path, then SparkSql rename the lower case path to the upper-case. while if the renamed partition contains more than one depth partition ,e.g. A=1/B=2, hive renamePartition change to a=1/b=2, then SparkSql rename it to A=1/B=2, but the a=1 still exists in the filesystem, we should also delete it. ## How was this patch tested? unit test added Author: windpiger <songjun@outlook.com> Closes #16700 from windpiger/clearUselessPathAfterRenamPartition.
Showing
- sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala 35 additions, 0 deletions...scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/PartitionProviderCompatibilitySuite.scala 36 additions, 0 deletions.../spark/sql/hive/PartitionProviderCompatibilitySuite.scala
Loading
Please register or sign in to comment