Skip to content
Snippets Groups Projects
Commit 1536d703 authored by zsxwing's avatar zsxwing Committed by Aaron Davidson
Browse files

[SPARK-4008] Fix "kryo with fold" in KryoSerializerSuite

`zeroValue` will be serialized by `spark.closure.serializer` but `spark.closure.serializer` only supports the default Java serializer. So it must not be `ClassWithoutNoArgConstructor`, which can not be serialized by the Java serializer.

This PR changed `zeroValue` to null and updated the test to make it work correctly.

Author: zsxwing <zsxwing@gmail.com>

Closes #2856 from zsxwing/SPARK-4008 and squashes the following commits:

51da655 [zsxwing] [SPARK-4008] Fix "kryo with fold" in KryoSerializerSuite
parent 84e5da87
No related branches found
No related tags found
No related merge requests found
......@@ -201,12 +201,17 @@ class KryoSerializerSuite extends FunSuite with SharedSparkContext {
assert(control.sum === result)
}
// TODO: this still doesn't work
ignore("kryo with fold") {
test("kryo with fold") {
val control = 1 :: 2 :: Nil
// zeroValue must not be a ClassWithoutNoArgConstructor instance because it will be
// serialized by spark.closure.serializer but spark.closure.serializer only supports
// the default Java serializer.
val result = sc.parallelize(control, 2).map(new ClassWithoutNoArgConstructor(_))
.fold(new ClassWithoutNoArgConstructor(10))((t1, t2) => new ClassWithoutNoArgConstructor(t1.x + t2.x)).x
assert(10 + control.sum === result)
.fold(null)((t1, t2) => {
val t1x = if (t1 == null) 0 else t1.x
new ClassWithoutNoArgConstructor(t1x + t2.x)
}).x
assert(control.sum === result)
}
test("kryo with nonexistent custom registrator should fail") {
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment