-
- Downloads
[SPARK-21204][SQL] Add support for Scala Set collection types in serialization
## What changes were proposed in this pull request? Currently we can't produce a `Dataset` containing `Set` in SparkSQL. This PR tries to support serialization/deserialization of `Set`. Because there's no corresponding internal data type in SparkSQL for a `Set`, the most proper choice for serializing a set should be an array. ## How was this patch tested? Added unit tests. Author: Liang-Chi Hsieh <viirya@gmail.com> Closes #18416 from viirya/SPARK-21204.
Showing
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala 25 additions, 3 deletions...scala/org/apache/spark/sql/catalyst/ScalaReflection.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala 3 additions, 2 deletions...ache/spark/sql/catalyst/expressions/objects/objects.scala
- sql/core/src/main/scala/org/apache/spark/sql/SQLImplicits.scala 10 additions, 0 deletions...re/src/main/scala/org/apache/spark/sql/SQLImplicits.scala
- sql/core/src/test/scala/org/apache/spark/sql/DataFrameAggregateSuite.scala 10 additions, 0 deletions.../scala/org/apache/spark/sql/DataFrameAggregateSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/DatasetPrimitiveSuite.scala 31 additions, 0 deletions...st/scala/org/apache/spark/sql/DatasetPrimitiveSuite.scala
Loading
Please register or sign in to comment