Skip to content
Snippets Groups Projects
Commit 9dc0ca06 authored by Jakob Odersky's avatar Jakob Odersky Committed by Michael Armbrust
Browse files

[SPARK-17368][SQL] Add support for value class serialization and deserialization

## What changes were proposed in this pull request?
Value classes were unsupported because catalyst data types were
obtained through reflection on erased types, which would resolve to a
value class' wrapped type and hence lead to unavailable methods during
code generation.

E.g. the following class
```scala
case class Foo(x: Int) extends AnyVal
```
would be seen as an `int` in catalyst and will cause instance cast failures when generated java code tries to treat it as a `Foo`.

This patch simply removes the erasure step when getting data types for
catalyst.

## How was this patch tested?
Additional tests in `ExpressionEncoderSuite`.

Author: Jakob Odersky <jakob@odersky.com>

Closes #15284 from jodersky/value-classes.
parent adc11242
No related branches found
No related tags found
No related merge requests found
......@@ -628,7 +628,7 @@ object ScalaReflection extends ScalaReflection {
/*
* Retrieves the runtime class corresponding to the provided type.
*/
def getClassFromType(tpe: Type): Class[_] = mirror.runtimeClass(tpe.erasure.typeSymbol.asClass)
def getClassFromType(tpe: Type): Class[_] = mirror.runtimeClass(tpe.typeSymbol.asClass)
case class Schema(dataType: DataType, nullable: Boolean)
......
......@@ -66,8 +66,6 @@ case class RepeatedData(
mapFieldNull: scala.collection.Map[Int, java.lang.Long],
structField: PrimitiveData)
case class SpecificCollection(l: List[Int])
/** For testing Kryo serialization based encoder. */
class KryoSerializable(val value: Int) {
override def hashCode(): Int = value
......@@ -107,6 +105,12 @@ class UDTForCaseClass extends UserDefinedType[UDTCaseClass] {
}
}
case class PrimitiveValueClass(wrapped: Int) extends AnyVal
case class ReferenceValueClass(wrapped: ReferenceValueClass.Container) extends AnyVal
object ReferenceValueClass {
case class Container(data: Int)
}
class ExpressionEncoderSuite extends PlanTest with AnalysisTest {
OuterScopes.addOuterScope(this)
......@@ -290,6 +294,12 @@ class ExpressionEncoderSuite extends PlanTest with AnalysisTest {
ExpressionEncoder.tuple(intEnc, ExpressionEncoder.tuple(intEnc, longEnc))
}
encodeDecodeTest(
PrimitiveValueClass(42), "primitive value class")
encodeDecodeTest(
ReferenceValueClass(ReferenceValueClass.Container(1)), "reference value class")
productTest(("UDT", new ExamplePoint(0.1, 0.2)))
test("nullable of encoder schema") {
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment