-
- Downloads
[SPARK-1845] [SQL] Use AllScalaRegistrar for SparkSqlSerializer to register serializers of ...
...Scala collections. When I execute `orderBy` or `limit` for `SchemaRDD` including `ArrayType` or `MapType`, `SparkSqlSerializer` throws the following exception: ``` com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): scala.collection.immutable.$colon$colon ``` or ``` com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): scala.collection.immutable.Vector ``` or ``` com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): scala.collection.immutable.HashMap$HashTrieMap ``` and so on. This is because registrations of serializers for each concrete collections are missing in `SparkSqlSerializer`. I believe it should use `AllScalaRegistrar`. `AllScalaRegistrar` covers a lot of serializers for concrete classes of `Seq`, `Map` for `ArrayType`, `MapType`. Author: Takuya UESHIN <ueshin@happy-camper.st> Closes #790 from ueshin/issues/SPARK-1845 and squashes the following commits: d1ed992 [Takuya UESHIN] Use AllScalaRegistrar for SparkSqlSerializer to register serializers of Scala collections.
Showing
- sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlSerializer.scala 2 additions, 26 deletions...a/org/apache/spark/sql/execution/SparkSqlSerializer.scala
- sql/core/src/test/scala/org/apache/spark/sql/DslQuerySuite.scala 24 additions, 0 deletions...e/src/test/scala/org/apache/spark/sql/DslQuerySuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala 30 additions, 0 deletions...e/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/TestData.scala 10 additions, 0 deletionssql/core/src/test/scala/org/apache/spark/sql/TestData.scala
Loading
Please register or sign in to comment