-
- Downloads
[SPARK-6052][SQL]In JSON schema inference, we should always set containsNull...
[SPARK-6052][SQL]In JSON schema inference, we should always set containsNull of an ArrayType to true Always set `containsNull = true` when infer the schema of JSON datasets. If we set `containsNull` based on records we scanned, we may miss arrays with null values when we do sampling. Also, because future data can have arrays with null values, if we convert JSON data to parquet, always setting `containsNull = true` is a more robust way to go. JIRA: https://issues.apache.org/jira/browse/SPARK-6052 Author: Yin Huai <yhuai@databricks.com> Closes #4806 from yhuai/jsonArrayContainsNull and squashes the following commits: 05eab9d [Yin Huai] Change containsNull to true.
Showing
- sql/core/src/main/scala/org/apache/spark/sql/json/JsonRDD.scala 4 additions, 5 deletions...re/src/main/scala/org/apache/spark/sql/json/JsonRDD.scala
- sql/core/src/test/scala/org/apache/spark/sql/json/JsonSuite.scala 19 additions, 19 deletions.../src/test/scala/org/apache/spark/sql/json/JsonSuite.scala
Loading
Please register or sign in to comment