Skip to content
  • Yuming Wang's avatar
    17edfec5
    [SPARK-20427][SQL] Read JDBC table use custom schema · 17edfec5
    Yuming Wang authored
    ## What changes were proposed in this pull request?
    
    Auto generated Oracle schema some times not we expect:
    
    - `number(1)` auto mapped to BooleanType, some times it's not we expect, per [SPARK-20921](https://issues.apache.org/jira/browse/SPARK-20921).
    -  `number` auto mapped to Decimal(38,10), It can't read big data, per [SPARK-20427](https://issues.apache.org/jira/browse/SPARK-20427).
    
    This PR fix this issue by custom schema as follows:
    ```scala
    val props = new Properties()
    props.put("customSchema", "ID decimal(38, 0), N1 int, N2 boolean")
    val dfRead = spark.read.schema(schema).jdbc(jdbcUrl, "tableWithCustomSchema", props)
    dfRead.show()
    ```
    or
    ```sql
    CREATE TEMPORARY VIEW tableWithCustomSchema
    USING org.apache.spark.sql.jdbc
    OPTIONS (url '$jdbcUrl', dbTable 'tableWithCustomSchema', customSchema'ID decimal(38, 0), N1 int, N2 boolean')
    ```
    
    ## How was this patch tested?
    
    unit tests
    
    Author: Yuming Wang <wgyumg@gmail.com>
    
    Closes #18266 from wangyum/SPARK-20427.
    17edfec5
    [SPARK-20427][SQL] Read JDBC table use custom schema
    Yuming Wang authored
    ## What changes were proposed in this pull request?
    
    Auto generated Oracle schema some times not we expect:
    
    - `number(1)` auto mapped to BooleanType, some times it's not we expect, per [SPARK-20921](https://issues.apache.org/jira/browse/SPARK-20921).
    -  `number` auto mapped to Decimal(38,10), It can't read big data, per [SPARK-20427](https://issues.apache.org/jira/browse/SPARK-20427).
    
    This PR fix this issue by custom schema as follows:
    ```scala
    val props = new Properties()
    props.put("customSchema", "ID decimal(38, 0), N1 int, N2 boolean")
    val dfRead = spark.read.schema(schema).jdbc(jdbcUrl, "tableWithCustomSchema", props)
    dfRead.show()
    ```
    or
    ```sql
    CREATE TEMPORARY VIEW tableWithCustomSchema
    USING org.apache.spark.sql.jdbc
    OPTIONS (url '$jdbcUrl', dbTable 'tableWithCustomSchema', customSchema'ID decimal(38, 0), N1 int, N2 boolean')
    ```
    
    ## How was this patch tested?
    
    unit tests
    
    Author: Yuming Wang <wgyumg@gmail.com>
    
    Closes #18266 from wangyum/SPARK-20427.
Loading