Skip to content
Snippets Groups Projects
Commit 251a9927 authored by Takuya UESHIN's avatar Takuya UESHIN Committed by Herman van Hovell
Browse files

[SPARK-18398][SQL] Fix nullabilities of MapObjects and ExternalMapToCatalyst.


## What changes were proposed in this pull request?

The nullabilities of `MapObject` can be made more strict by relying on `inputObject.nullable` and `lambdaFunction.nullable`.

Also `ExternalMapToCatalyst.dataType` can be made more strict by relying on `valueConverter.nullable`.

## How was this patch tested?

Existing tests.

Author: Takuya UESHIN <ueshin@happy-camper.st>

Closes #15840 from ueshin/issues/SPARK-18398.

(cherry picked from commit 9f262ae1)
Signed-off-by: default avatarHerman van Hovell <hvanhovell@databricks.com>
parent 31002e4a
No related branches found
No related tags found
No related merge requests found
......@@ -461,14 +461,15 @@ case class MapObjects private(
lambdaFunction: Expression,
inputData: Expression) extends Expression with NonSQLExpression {
override def nullable: Boolean = true
override def nullable: Boolean = inputData.nullable
override def children: Seq[Expression] = lambdaFunction :: inputData :: Nil
override def eval(input: InternalRow): Any =
throw new UnsupportedOperationException("Only code-generated evaluation is supported")
override def dataType: DataType = ArrayType(lambdaFunction.dataType)
override def dataType: DataType =
ArrayType(lambdaFunction.dataType, containsNull = lambdaFunction.nullable)
override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
val elementJavaType = ctx.javaType(loopVarDataType)
......@@ -642,7 +643,8 @@ case class ExternalMapToCatalyst private(
override def foldable: Boolean = false
override def dataType: MapType = MapType(keyConverter.dataType, valueConverter.dataType)
override def dataType: MapType = MapType(
keyConverter.dataType, valueConverter.dataType, valueContainsNull = valueConverter.nullable)
override def eval(input: InternalRow): Any =
throw new UnsupportedOperationException("Only code-generated evaluation is supported")
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment