-
- Downloads
[SPARK-12057][SQL] Prevent failure on corrupt JSON records
This PR makes JSON parser and schema inference handle more cases where we have unparsed records. It is based on #10043. The last commit fixes the failed test and updates the logic of schema inference. Regarding the schema inference change, if we have something like ``` {"f1":1} [1,2,3] ``` originally, we will get a DF without any column. After this change, we will get a DF with columns `f1` and `_corrupt_record`. Basically, for the second row, `[1,2,3]` will be the value of `_corrupt_record`. When merge this PR, please make sure that the author is simplyianm. JIRA: https://issues.apache.org/jira/browse/SPARK-12057 Closes #10043 Author: Ian Macalinao <me@ian.pw> Author: Yin Huai <yhuai@databricks.com> Closes #10288 from yhuai/handleCorruptJson.
Showing
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/InferSchema.scala 33 additions, 4 deletions...he/spark/sql/execution/datasources/json/InferSchema.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JacksonParser.scala 12 additions, 7 deletions.../spark/sql/execution/datasources/json/JacksonParser.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala 37 additions, 0 deletions...ache/spark/sql/execution/datasources/json/JsonSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/TestJsonData.scala 8 additions, 1 deletion...e/spark/sql/execution/datasources/json/TestJsonData.scala
Loading
Please register or sign in to comment