Skip to content
Snippets Groups Projects
Commit e374b242 authored by Eric Liang's avatar Eric Liang Committed by Wenchen Fan
Browse files

[SPARK-18659][SQL] Incorrect behaviors in overwrite table for datasource tables


## What changes were proposed in this pull request?

Two bugs are addressed here
1. INSERT OVERWRITE TABLE sometime crashed when catalog partition management was enabled. This was because when dropping partitions after an overwrite operation, the Hive client will attempt to delete the partition files. If the entire partition directory was dropped, this would fail. The PR fixes this by adding a flag to control whether the Hive client should attempt to delete files.
2. The static partition spec for OVERWRITE TABLE was not correctly resolved to the case-sensitive original partition names. This resulted in the entire table being overwritten if you did not correctly capitalize your partition names.

cc yhuai cloud-fan

## How was this patch tested?

Unit tests. Surprisingly, the existing overwrite table tests did not catch these edge cases.

Author: Eric Liang <ekl@databricks.com>

Closes #16088 from ericl/spark-18659.

(cherry picked from commit 7935c847)
Signed-off-by: default avatarWenchen Fan <wenchen@databricks.com>
parent 415730e1
No related branches found
No related tags found
No related merge requests found
Showing
with 110 additions and 37 deletions
Loading
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment