-
- Downloads
[SPARK-7180] [SPARK-8090] [SPARK-8091] Fix a number of SerializationDebugger bugs and limitations
This PR solves three SerializationDebugger issues. * SPARK-7180 - SerializationDebugger fails with ArrayOutOfBoundsException * SPARK-8090 - SerializationDebugger does not handle classes with writeReplace correctly * SPARK-8091 - SerializationDebugger does not handle classes with writeObject method The solutions for each are explained as follows * SPARK-7180 - The wrong slot desc was used for getting the value of the fields in the object being tested. * SPARK-8090 - Test the type of the replaced object. * SPARK-8091 - Use a dummy ObjectOutputStream to collect all the objects written by the writeObject() method, and then test those objects as usual. I also added more tests in the testsuite to increase code coverage. For example, added tests for cases where there are not serializability issues. Author: Tathagata Das <tathagata.das1565@gmail.com> Closes #6625 from tdas/SPARK-7180 and squashes the following commits: c7cb046 [Tathagata Das] Addressed comments on docs ae212c8 [Tathagata Das] Improved docs 304c97b [Tathagata Das] Fixed build error 26b5179 [Tathagata Das] more tests.....92% line coverage 7e2fdcf [Tathagata Das] Added more tests d1967fb [Tathagata Das] Added comments. da75d34 [Tathagata Das] Removed unnecessary lines. 50a608d [Tathagata Das] Fixed bugs and added support for writeObject
Showing
- core/src/main/scala/org/apache/spark/serializer/SerializationDebugger.scala 106 additions, 6 deletions...a/org/apache/spark/serializer/SerializationDebugger.scala
- core/src/test/scala/org/apache/spark/serializer/SerializationDebuggerSuite.scala 115 additions, 4 deletions.../apache/spark/serializer/SerializationDebuggerSuite.scala
- streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala 2 additions, 2 deletions...n/scala/org/apache/spark/streaming/StreamingContext.scala
Loading
Please register or sign in to comment