-
- Downloads
[SPARK-11745][SQL] Enable more JSON parsing options
This patch adds the following options to the JSON data source, for dealing with non-standard JSON files: * `allowComments` (default `false`): ignores Java/C++ style comment in JSON records * `allowUnquotedFieldNames` (default `false`): allows unquoted JSON field names * `allowSingleQuotes` (default `true`): allows single quotes in addition to double quotes * `allowNumericLeadingZeros` (default `false`): allows leading zeros in numbers (e.g. 00012) To avoid passing a lot of options throughout the json package, I introduced a new JSONOptions case class to define all JSON config options. Also updated documentation to explain these options. Scala  Python  Author: Reynold Xin <rxin@databricks.com> Closes #9724 from rxin/SPARK-11745.
Showing
- python/pyspark/sql/readwriter.py 10 additions, 0 deletionspython/pyspark/sql/readwriter.py
- sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala 14 additions, 8 deletions...src/main/scala/org/apache/spark/sql/DataFrameReader.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/SparkPlan.scala 1 addition, 16 deletions...main/scala/org/apache/spark/sql/execution/SparkPlan.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/InferSchema.scala 19 additions, 15 deletions...he/spark/sql/execution/datasources/json/InferSchema.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JSONOptions.scala 64 additions, 0 deletions...he/spark/sql/execution/datasources/json/JSONOptions.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JSONRelation.scala 8 additions, 12 deletions...e/spark/sql/execution/datasources/json/JSONRelation.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JacksonParser.scala 43 additions, 39 deletions.../spark/sql/execution/datasources/json/JacksonParser.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonParsingOptionsSuite.scala 114 additions, 0 deletions.../execution/datasources/json/JsonParsingOptionsSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala 13 additions, 16 deletions...ache/spark/sql/execution/datasources/json/JsonSuite.scala
Loading
Please register or sign in to comment