-
- Downloads
[SPARK-13764][SQL] Parse modes in JSON data source
## What changes were proposed in this pull request? Currently, there is no way to control the behaviour when fails to parse corrupt records in JSON data source . This PR adds the support for parse modes just like CSV data source. There are three modes below: - `PERMISSIVE` : When it fails to parse, this sets `null` to to field. This is a default mode when it has been this mode. - `DROPMALFORMED`: When it fails to parse, this drops the whole record. - `FAILFAST`: When it fails to parse, it just throws an exception. This PR also make JSON data source share the `ParseModes` in CSV data source. ## How was this patch tested? Unit tests were used and `./dev/run_tests` for code style tests. Author: hyukjinkwon <gurwls223@gmail.com> Closes #11756 from HyukjinKwon/SPARK-13764.
Showing
- python/pyspark/sql/readwriter.py 8 additions, 0 deletionspython/pyspark/sql/readwriter.py
- sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala 18 additions, 0 deletions...src/main/scala/org/apache/spark/sql/DataFrameReader.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/ParseModes.scala 41 additions, 0 deletions...g/apache/spark/sql/execution/datasources/ParseModes.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVOptions.scala 2 additions, 25 deletions...ache/spark/sql/execution/datasources/csv/CSVOptions.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/InferSchema.scala 16 additions, 10 deletions...he/spark/sql/execution/datasources/json/InferSchema.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JSONOptions.scala 13 additions, 2 deletions...he/spark/sql/execution/datasources/json/JSONOptions.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JacksonParser.scala 15 additions, 7 deletions.../spark/sql/execution/datasources/json/JacksonParser.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala 51 additions, 1 deletion...ache/spark/sql/execution/datasources/json/JsonSuite.scala
Loading
Please register or sign in to comment