-
- Downloads
[SPARK-21648][SQL] Fix confusing assert failure in JDBC source when parallel...
[SPARK-21648][SQL] Fix confusing assert failure in JDBC source when parallel fetching parameters are not properly provided. ### What changes were proposed in this pull request? ```SQL CREATE TABLE mytesttable1 USING org.apache.spark.sql.jdbc OPTIONS ( url 'jdbc:mysql://${jdbcHostname}:${jdbcPort}/${jdbcDatabase}?user=${jdbcUsername}&password=${jdbcPassword}', dbtable 'mytesttable1', paritionColumn 'state_id', lowerBound '0', upperBound '52', numPartitions '53', fetchSize '10000' ) ``` The above option name `paritionColumn` is wrong. That mean, users did not provide the value for `partitionColumn`. In such case, users hit a confusing error. ``` AssertionError: assertion failed java.lang.AssertionError: assertion failed at scala.Predef$.assert(Predef.scala:156) at org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.createRelation(JdbcRelationProvider.scala:39) at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:312) ``` ### How was this patch tested? Added a test case Author: gatorsmile <gatorsmile@gmail.com> Closes #18864 from gatorsmile/jdbcPartCol.
Showing
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCOptions.scala 7 additions, 4 deletions...he/spark/sql/execution/datasources/jdbc/JDBCOptions.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcRelationProvider.scala 7 additions, 2 deletions...sql/execution/datasources/jdbc/JdbcRelationProvider.scala
- sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala 22 additions, 0 deletions.../src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCWriteSuite.scala 3 additions, 2 deletions...test/scala/org/apache/spark/sql/jdbc/JDBCWriteSuite.scala
Loading
Please register or sign in to comment