-
- Downloads
[SPARK-17719][SPARK-17776][SQL] Unify and tie up options in a single place in...
[SPARK-17719][SPARK-17776][SQL] Unify and tie up options in a single place in JDBC datasource package ## What changes were proposed in this pull request? This PR proposes to fix arbitrary usages among `Map[String, String]`, `Properties` and `JDBCOptions` instances for options in `execution/jdbc` package and make the connection properties exclude Spark-only options. This PR includes some changes as below: - Unify `Map[String, String]`, `Properties` and `JDBCOptions` in `execution/jdbc` package to `JDBCOptions`. - Move `batchsize`, `fetchszie`, `driver` and `isolationlevel` options into `JDBCOptions` instance. - Document `batchSize` and `isolationlevel` with marking both read-only options and write-only options. Also, this includes minor types and detailed explanation for some statements such as url. - Throw exceptions fast by checking arguments first rather than in execution time (e.g. for `fetchsize`). - Exclude Spark-only options in connection properties. ## How was this patch tested? Existing tests should cover this. Author: hyukjinkwon <gurwls223@gmail.com> Closes #15292 from HyukjinKwon/SPARK-17719.
Showing
- docs/sql-programming-guide.md 27 additions, 9 deletionsdocs/sql-programming-guide.md
- sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala 5 additions, 8 deletions...src/main/scala/org/apache/spark/sql/DataFrameReader.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCOptions.scala 95 additions, 15 deletions...he/spark/sql/execution/datasources/jdbc/JDBCOptions.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRDD.scala 17 additions, 28 deletions...apache/spark/sql/execution/datasources/jdbc/JDBCRDD.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRelation.scala 8 additions, 12 deletions...e/spark/sql/execution/datasources/jdbc/JDBCRelation.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcRelationProvider.scala 11 additions, 19 deletions...sql/execution/datasources/jdbc/JdbcRelationProvider.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala 8 additions, 34 deletions...ache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala
- sql/core/src/main/scala/org/apache/spark/sql/jdbc/PostgresDialect.scala 2 additions, 2 deletions...ain/scala/org/apache/spark/sql/jdbc/PostgresDialect.scala
- sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala 5 additions, 6 deletions.../src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCWriteSuite.scala 4 additions, 4 deletions...test/scala/org/apache/spark/sql/jdbc/JDBCWriteSuite.scala
Loading
Please register or sign in to comment