Skip to content
Snippets Groups Projects
Commit 3ced39df authored by Wenchen Fan's avatar Wenchen Fan
Browse files

[SPARK-17432][SQL] PreprocessDDL should respect case sensitivity when checking duplicated columns

## What changes were proposed in this pull request?

In `PreprocessDDL` we will check if table columns are duplicated. However, this checking ignores case sensitivity config(it's always case-sensitive) and lead to different result between `HiveExternalCatalog` and `InMemoryCatalog`. `HiveExternalCatalog` will throw exception because hive metastore is always case-nonsensitive, and `InMemoryCatalog` is fine.

This PR fixes it.

## How was this patch tested?

a new test in DDLSuite

Author: Wenchen Fan <wenchen@databricks.com>

Closes #14994 from cloud-fan/check-dup.
parent b230fb92
No related branches found
No related tags found
No related merge requests found
......@@ -97,7 +97,12 @@ case class PreprocessDDL(conf: SQLConf) extends Rule[LogicalPlan] {
// * sort columns' type must be orderable.
case c @ CreateTable(tableDesc, mode, query) if c.childrenResolved =>
val schema = if (query.isDefined) query.get.schema else tableDesc.schema
checkDuplication(schema.map(_.name), "table definition of " + tableDesc.identifier)
val columnNames = if (conf.caseSensitiveAnalysis) {
schema.map(_.name)
} else {
schema.map(_.name.toLowerCase)
}
checkDuplication(columnNames, "table definition of " + tableDesc.identifier)
val partitionColsChecked = checkPartitionColumns(schema, tableDesc)
val bucketColsChecked = checkBucketColumns(schema, partitionColsChecked)
......
......@@ -371,6 +371,13 @@ class DDLSuite extends QueryTest with SharedSQLContext with BeforeAndAfterEach {
sql("CREATE TABLE tbl(a int, a string) USING json")
}
assert(e.message == "Found duplicate column(s) in table definition of `tbl`: a")
withSQLConf(SQLConf.CASE_SENSITIVE.key -> "false") {
val e2 = intercept[AnalysisException] {
sql("CREATE TABLE tbl(a int, A string) USING json")
}
assert(e2.message == "Found duplicate column(s) in table definition of `tbl`: a")
}
}
test("create table - partition column names not in table definition") {
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment