-
- Downloads
[SPARK-19318][SQL] Fix to treat JDBC connection properties specified by the...
[SPARK-19318][SQL] Fix to treat JDBC connection properties specified by the user in case-sensitive manner. ## What changes were proposed in this pull request? The reason for test failure is that the property “oracle.jdbc.mapDateToTimestamp” set by the test was getting converted into all lower case. Oracle database expects this property in case-sensitive manner. This test was passing in previous releases because connection properties were sent as user specified for the test case scenario. Fixes to handle all option uniformly in case-insensitive manner, converted the JDBC connection properties also to lower case. This PR enhances CaseInsensitiveMap to keep track of input case-sensitive keys , and uses those when creating connection properties that are passed to the JDBC connection. Alternative approach PR https://github.com/apache/spark/pull/16847 is to pass original input keys to JDBC data source by adding check in the Data source class and handle case-insensitivity in the JDBC source code. ## How was this patch tested? Added new test cases to JdbcSuite , and OracleIntegrationSuite. Ran docker integration tests passed on my laptop, all tests passed successfully. Author: sureshthalamati <suresh.thalamati@gmail.com> Closes #16891 from sureshthalamati/jdbc_case_senstivity_props_fix-SPARK-19318.
Showing
- external/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/OracleIntegrationSuite.scala 36 additions, 0 deletions...la/org/apache/spark/sql/jdbc/OracleIntegrationSuite.scala
- mllib/src/main/scala/org/apache/spark/ml/source/libsvm/LibSVMOptions.scala 2 additions, 2 deletions...ala/org/apache/spark/ml/source/libsvm/LibSVMOptions.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JSONOptions.scala 2 additions, 2 deletions...cala/org/apache/spark/sql/catalyst/json/JSONOptions.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/CaseInsensitiveMap.scala 23 additions, 9 deletions...g/apache/spark/sql/catalyst/util/CaseInsensitiveMap.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala 2 additions, 2 deletions...g/apache/spark/sql/execution/datasources/DataSource.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVOptions.scala 3 additions, 3 deletions...ache/spark/sql/execution/datasources/csv/CSVOptions.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCOptions.scala 5 additions, 5 deletions...he/spark/sql/execution/datasources/jdbc/JDBCOptions.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetOptions.scala 2 additions, 2 deletions...rk/sql/execution/datasources/parquet/ParquetOptions.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/text/TextOptions.scala 2 additions, 2 deletions...he/spark/sql/execution/datasources/text/TextOptions.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamOptions.scala 2 additions, 2 deletions...che/spark/sql/execution/streaming/FileStreamOptions.scala
- sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala 49 additions, 1 deletion.../src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCWriteSuite.scala 13 additions, 0 deletions...test/scala/org/apache/spark/sql/jdbc/JDBCWriteSuite.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala 2 additions, 2 deletions...scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/HiveOptions.scala 3 additions, 2 deletions...ala/org/apache/spark/sql/hive/execution/HiveOptions.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcOptions.scala 2 additions, 2 deletions...main/scala/org/apache/spark/sql/hive/orc/OrcOptions.scala
Loading
Please register or sign in to comment