-
- Downloads
[SPARK-20125][SQL] Dataset of type option of map does not work
## What changes were proposed in this pull request? When we build the deserializer expression for map type, we will use `StaticInvoke` to call `ArrayBasedMapData.toScalaMap`, and declare the return type as `scala.collection.immutable.Map`. If the map is inside an Option, we will wrap this `StaticInvoke` with `WrapOption`, which requires the input to be `scala.collect.Map`. Ideally this should be fine, as `scala.collection.immutable.Map` extends `scala.collect.Map`, but our `ObjectType` is too strict about this, this PR fixes it. ## How was this patch tested? new regression test Author: Wenchen Fan <wenchen@databricks.com> Closes #17454 from cloud-fan/map.
Showing
- sql/catalyst/src/main/scala/org/apache/spark/sql/types/ObjectType.scala 5 additions, 0 deletions...rc/main/scala/org/apache/spark/sql/types/ObjectType.scala
- sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala 6 additions, 0 deletions...re/src/test/scala/org/apache/spark/sql/DatasetSuite.scala
Please register or sign in to comment