-
- Downloads
[SPARK-18433][SQL] Improve DataSource option keys to be more case-insensitive
## What changes were proposed in this pull request? This PR aims to improve DataSource option keys to be more case-insensitive DataSource partially use CaseInsensitiveMap in code-path. For example, the following fails to find url. ```scala val df = spark.createDataFrame(sparkContext.parallelize(arr2x2), schema2) df.write.format("jdbc") .option("UrL", url1) .option("dbtable", "TEST.SAVETEST") .options(properties.asScala) .save() ``` This PR makes DataSource options to use CaseInsensitiveMap internally and also makes DataSource to use CaseInsensitiveMap generally except `InMemoryFileIndex` and `InsertIntoHadoopFsRelationCommand`. We can not pass them CaseInsensitiveMap because they creates new case-sensitive HadoopConfs by calling newHadoopConfWithOptions(options) inside. ## How was this patch tested? Pass the Jenkins test with newly added test cases. Author: Dongjoon Hyun <dongjoon@apache.org> Closes #15884 from dongjoon-hyun/SPARK-18433. (cherry picked from commit 74f5c217) Signed-off-by:Wenchen Fan <wenchen@databricks.com>
Showing
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JSONOptions.scala 4 additions, 2 deletions...cala/org/apache/spark/sql/catalyst/json/JSONOptions.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/CaseInsensitiveMap.scala 36 additions, 0 deletions...g/apache/spark/sql/catalyst/util/CaseInsensitiveMap.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala 1 addition, 1 deletion...in/scala/org/apache/spark/sql/execution/command/ddl.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala 16 additions, 14 deletions...g/apache/spark/sql/execution/datasources/DataSource.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVOptions.scala 5 additions, 3 deletions...ache/spark/sql/execution/datasources/csv/CSVOptions.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/ddl.scala 0 additions, 18 deletions...cala/org/apache/spark/sql/execution/datasources/ddl.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCOptions.scala 7 additions, 3 deletions...he/spark/sql/execution/datasources/jdbc/JDBCOptions.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetOptions.scala 5 additions, 1 deletion...rk/sql/execution/datasources/parquet/ParquetOptions.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamOptions.scala 5 additions, 3 deletions...che/spark/sql/execution/streaming/FileStreamOptions.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVInferSchemaSuite.scala 5 additions, 0 deletions...k/sql/execution/datasources/csv/CSVInferSchemaSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala 17 additions, 2 deletions...ache/spark/sql/execution/datasources/json/JsonSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetIOSuite.scala 7 additions, 0 deletions...rk/sql/execution/datasources/parquet/ParquetIOSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCWriteSuite.scala 9 additions, 0 deletions...test/scala/org/apache/spark/sql/jdbc/JDBCWriteSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/streaming/FileStreamSourceSuite.scala 5 additions, 0 deletions...rg/apache/spark/sql/streaming/FileStreamSourceSuite.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala 1 addition, 1 deletion...scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcOptions.scala 5 additions, 1 deletion...main/scala/org/apache/spark/sql/hive/orc/OrcOptions.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/OrcSourceSuite.scala 4 additions, 0 deletions.../scala/org/apache/spark/sql/hive/orc/OrcSourceSuite.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/parquetSuites.scala 1 addition, 0 deletions.../test/scala/org/apache/spark/sql/hive/parquetSuites.scala
Loading
Please register or sign in to comment