Skip to content
Snippets Groups Projects
Commit 7b5ea000 authored by Burak Yavuz's avatar Burak Yavuz Committed by Herman van Hovell
Browse files

[SPARK-19543] from_json fails when the input row is empty


## What changes were proposed in this pull request?

Using from_json on a column with an empty string results in: java.util.NoSuchElementException: head of empty list.

This is because `parser.parse(input)` may return `Nil` when `input.trim.isEmpty`

## How was this patch tested?

Regression test in `JsonExpressionsSuite`

Author: Burak Yavuz <brkyvz@gmail.com>

Closes #16881 from brkyvz/json-fix.

(cherry picked from commit d5593f7f)
Signed-off-by: default avatarHerman van Hovell <hvanhovell@databricks.com>
parent ff5818b8
No related branches found
No related tags found
No related merge requests found
......@@ -497,7 +497,7 @@ case class JsonToStruct(schema: StructType, options: Map[String, String], child:
override def dataType: DataType = schema
override def nullSafeEval(json: Any): Any = {
try parser.parse(json.toString).head catch {
try parser.parse(json.toString).headOption.orNull catch {
case _: SparkSQLJsonProcessingException => null
}
}
......
......@@ -376,6 +376,14 @@ class JsonExpressionsSuite extends SparkFunSuite with ExpressionEvalHelper {
)
}
test("SPARK-19543: from_json empty input column") {
val schema = StructType(StructField("a", IntegerType) :: Nil)
checkEvaluation(
JsonToStruct(schema, Map.empty, Literal.create(" ", StringType)),
null
)
}
test("to_json") {
val schema = StructType(StructField("a", IntegerType) :: Nil)
val struct = Literal.create(create_row(1), schema)
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment