Skip to content
Snippets Groups Projects
  • hyukjinkwon's avatar
    e4740881
    [SPARK-13764][SQL] Parse modes in JSON data source · e4740881
    hyukjinkwon authored
    ## What changes were proposed in this pull request?
    
    Currently, there is no way to control the behaviour when fails to parse corrupt records in JSON data source .
    
    This PR adds the support for parse modes just like CSV data source. There are three modes below:
    
    - `PERMISSIVE` :  When it fails to parse, this sets `null` to to field. This is a default mode when it has been this mode.
    - `DROPMALFORMED`: When it fails to parse, this drops the whole record.
    - `FAILFAST`: When it fails to parse, it just throws an exception.
    
    This PR also make JSON data source share the `ParseModes` in CSV data source.
    
    ## How was this patch tested?
    
    Unit tests were used and `./dev/run_tests` for code style tests.
    
    Author: hyukjinkwon <gurwls223@gmail.com>
    
    Closes #11756 from HyukjinKwon/SPARK-13764.
    e4740881
    History
    [SPARK-13764][SQL] Parse modes in JSON data source
    hyukjinkwon authored
    ## What changes were proposed in this pull request?
    
    Currently, there is no way to control the behaviour when fails to parse corrupt records in JSON data source .
    
    This PR adds the support for parse modes just like CSV data source. There are three modes below:
    
    - `PERMISSIVE` :  When it fails to parse, this sets `null` to to field. This is a default mode when it has been this mode.
    - `DROPMALFORMED`: When it fails to parse, this drops the whole record.
    - `FAILFAST`: When it fails to parse, it just throws an exception.
    
    This PR also make JSON data source share the `ParseModes` in CSV data source.
    
    ## How was this patch tested?
    
    Unit tests were used and `./dev/run_tests` for code style tests.
    
    Author: hyukjinkwon <gurwls223@gmail.com>
    
    Closes #11756 from HyukjinKwon/SPARK-13764.