-
- Downloads
SPARK-897: preemptively serialize closures
These commits cause `ClosureCleaner.clean` to attempt to serialize the cleaned closure with the default closure serializer and throw a `SparkException` if doing so fails. This behavior is enabled by default but can be disabled at individual callsites of `SparkContext.clean`. Commit 98e01ae8 fixes some no-op assertions in `GraphSuite` that this work exposed; I'm happy to put that in a separate PR if that would be more appropriate. Author: William Benton <willb@redhat.com> Closes #143 from willb/spark-897 and squashes the following commits: bceab8a [William Benton] Commented DStream corner cases for serializability checking. 64d04d2 [William Benton] FailureSuite now checks both messages and causes. 3b3f74a [William Benton] Stylistic and doc cleanups. b215dea [William Benton] Fixed spurious failures in ImplicitOrderingSuite be1ecd6 [William Benton] Don't check serializability of DStream transforms. abe816b [William Benton] Make proactive serializability checking optional. 5bfff24 [William Benton] Adds proactive closure-serializablilty checking ed2ccf0 [William Benton] Test cases for SPARK-897.
Showing
- core/src/main/scala/org/apache/spark/SparkContext.scala 10 additions, 2 deletionscore/src/main/scala/org/apache/spark/SparkContext.scala
- core/src/main/scala/org/apache/spark/util/ClosureCleaner.scala 14 additions, 2 deletions...src/main/scala/org/apache/spark/util/ClosureCleaner.scala
- core/src/test/scala/org/apache/spark/FailureSuite.scala 10 additions, 4 deletionscore/src/test/scala/org/apache/spark/FailureSuite.scala
- core/src/test/scala/org/apache/spark/ImplicitOrderingSuite.scala 52 additions, 23 deletions...c/test/scala/org/apache/spark/ImplicitOrderingSuite.scala
- core/src/test/scala/org/apache/spark/serializer/ProactiveClosureSerializationSuite.scala 90 additions, 0 deletions...spark/serializer/ProactiveClosureSerializationSuite.scala
- streaming/src/main/scala/org/apache/spark/streaming/dstream/DStream.scala 20 additions, 5 deletions...in/scala/org/apache/spark/streaming/dstream/DStream.scala
Loading
Please register or sign in to comment