-
- Downloads
[SPARK-2969][SQL] Make ScalaReflection be able to handle...
[SPARK-2969][SQL] Make ScalaReflection be able to handle ArrayType.containsNull and MapType.valueContainsNull. Make `ScalaReflection` be able to handle like: - `Seq[Int]` as `ArrayType(IntegerType, containsNull = false)` - `Seq[java.lang.Integer]` as `ArrayType(IntegerType, containsNull = true)` - `Map[Int, Long]` as `MapType(IntegerType, LongType, valueContainsNull = false)` - `Map[Int, java.lang.Long]` as `MapType(IntegerType, LongType, valueContainsNull = true)` Author: Takuya UESHIN <ueshin@happy-camper.st> Closes #1889 from ueshin/issues/SPARK-2969 and squashes the following commits: 24f1c5c [Takuya UESHIN] Change the default value of ArrayType.containsNull to true in Python API. 79f5b65 [Takuya UESHIN] Change the default value of ArrayType.containsNull to true in Java API. 7cd1a7a [Takuya UESHIN] Fix json test failures. 2cfb862 [Takuya UESHIN] Change the default value of ArrayType.containsNull to true. 2f38e61 [Takuya UESHIN] Revert the default value of MapTypes.valueContainsNull. 9fa02f5 [Takuya UESHIN] Fix a test failure. 1a9a96b [Takuya UESHIN] Modify ScalaReflection to handle ArrayType.containsNull and MapType.valueContainsNull.
Showing
- python/pyspark/sql.py 3 additions, 3 deletionspython/pyspark/sql.py
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala 6 additions, 3 deletions...scala/org/apache/spark/sql/catalyst/ScalaReflection.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/types/dataTypes.scala 2 additions, 2 deletions...scala/org/apache/spark/sql/catalyst/types/dataTypes.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/ScalaReflectionSuite.scala 19 additions, 3 deletions.../org/apache/spark/sql/catalyst/ScalaReflectionSuite.scala
- sql/core/src/main/java/org/apache/spark/sql/api/java/DataType.java 2 additions, 2 deletions...src/main/java/org/apache/spark/sql/api/java/DataType.java
- sql/core/src/test/scala/org/apache/spark/sql/DataTypeSuite.scala 1 addition, 1 deletion...e/src/test/scala/org/apache/spark/sql/DataTypeSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/json/JsonSuite.scala 16 additions, 16 deletions.../src/test/scala/org/apache/spark/sql/json/JsonSuite.scala
Loading
Please register or sign in to comment