Skip to content
  • Takuya UESHIN's avatar
    98c2bb0b
    [SPARK-2969][SQL] Make ScalaReflection be able to handle... · 98c2bb0b
    Takuya UESHIN authored
    [SPARK-2969][SQL] Make ScalaReflection be able to handle ArrayType.containsNull and MapType.valueContainsNull.
    
    Make `ScalaReflection` be able to handle like:
    
    - `Seq[Int]` as `ArrayType(IntegerType, containsNull = false)`
    - `Seq[java.lang.Integer]` as `ArrayType(IntegerType, containsNull = true)`
    - `Map[Int, Long]` as `MapType(IntegerType, LongType, valueContainsNull = false)`
    - `Map[Int, java.lang.Long]` as `MapType(IntegerType, LongType, valueContainsNull = true)`
    
    Author: Takuya UESHIN <ueshin@happy-camper.st>
    
    Closes #1889 from ueshin/issues/SPARK-2969 and squashes the following commits:
    
    24f1c5c [Takuya UESHIN] Change the default value of ArrayType.containsNull to true in Python API.
    79f5b65 [Takuya UESHIN] Change the default value of ArrayType.containsNull to true in Java API.
    7cd1a7a [Takuya UESHIN] Fix json test failures.
    2cfb862 [Takuya UESHIN] Change the default value of ArrayType.containsNull to true.
    2f38e61 [Takuya UESHIN] Revert the default value of MapTypes.valueContainsNull.
    9fa02f5 [Takuya UESHIN] Fix a test failure.
    1a9a96b [Takuya UESHIN] Modify ScalaReflection to handle ArrayType.containsNull and MapType.valueContainsNull.
    98c2bb0b
    [SPARK-2969][SQL] Make ScalaReflection be able to handle...
    Takuya UESHIN authored
    [SPARK-2969][SQL] Make ScalaReflection be able to handle ArrayType.containsNull and MapType.valueContainsNull.
    
    Make `ScalaReflection` be able to handle like:
    
    - `Seq[Int]` as `ArrayType(IntegerType, containsNull = false)`
    - `Seq[java.lang.Integer]` as `ArrayType(IntegerType, containsNull = true)`
    - `Map[Int, Long]` as `MapType(IntegerType, LongType, valueContainsNull = false)`
    - `Map[Int, java.lang.Long]` as `MapType(IntegerType, LongType, valueContainsNull = true)`
    
    Author: Takuya UESHIN <ueshin@happy-camper.st>
    
    Closes #1889 from ueshin/issues/SPARK-2969 and squashes the following commits:
    
    24f1c5c [Takuya UESHIN] Change the default value of ArrayType.containsNull to true in Python API.
    79f5b65 [Takuya UESHIN] Change the default value of ArrayType.containsNull to true in Java API.
    7cd1a7a [Takuya UESHIN] Fix json test failures.
    2cfb862 [Takuya UESHIN] Change the default value of ArrayType.containsNull to true.
    2f38e61 [Takuya UESHIN] Revert the default value of MapTypes.valueContainsNull.
    9fa02f5 [Takuya UESHIN] Fix a test failure.
    1a9a96b [Takuya UESHIN] Modify ScalaReflection to handle ArrayType.containsNull and MapType.valueContainsNull.
Loading