-
- Downloads
[SPARK-18850][SS] Make StreamExecution and progress classes serializable
## What changes were proposed in this pull request? This PR adds StreamingQueryWrapper to make StreamExecution and progress classes serializable because it is too easy for it to get captured with normal usage. If StreamingQueryWrapper gets captured in a closure but no place calls its methods, it should not fail the Spark tasks. However if its methods are called, then this PR will throw a better message. ## How was this patch tested? `test("StreamingQuery should be Serializable but cannot be used in executors")` `test("progress classes should be Serializable")` Author: Shixiong Zhu <shixiong@databricks.com> Closes #16272 from zsxwing/SPARK-18850.
Showing
- sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/ProgressReporter.scala 2 additions, 2 deletions...ache/spark/sql/execution/streaming/ProgressReporter.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamingQueryWrapper.scala 107 additions, 0 deletions...spark/sql/execution/streaming/StreamingQueryWrapper.scala
- sql/core/src/main/scala/org/apache/spark/sql/streaming/StreamingQueryManager.scala 4 additions, 4 deletions...rg/apache/spark/sql/streaming/StreamingQueryManager.scala
- sql/core/src/main/scala/org/apache/spark/sql/streaming/StreamingQueryStatus.scala 5 additions, 1 deletion...org/apache/spark/sql/streaming/StreamingQueryStatus.scala
- sql/core/src/main/scala/org/apache/spark/sql/streaming/progress.scala 4 additions, 4 deletions.../main/scala/org/apache/spark/sql/streaming/progress.scala
- sql/core/src/test/scala/org/apache/spark/sql/streaming/FileStreamSourceSuite.scala 4 additions, 2 deletions...rg/apache/spark/sql/streaming/FileStreamSourceSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamSuite.scala 2 additions, 2 deletions...st/scala/org/apache/spark/sql/streaming/StreamSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamTest.scala 2 additions, 1 deletion...est/scala/org/apache/spark/sql/streaming/StreamTest.scala
- sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQueryManagerSuite.scala 2 additions, 3 deletions...ache/spark/sql/streaming/StreamingQueryManagerSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQueryStatusAndProgressSuite.scala 45 additions, 7 deletions.../sql/streaming/StreamingQueryStatusAndProgressSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQuerySuite.scala 43 additions, 1 deletion.../org/apache/spark/sql/streaming/StreamingQuerySuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/streaming/test/DataStreamReaderWriterSuite.scala 2 additions, 2 deletions...park/sql/streaming/test/DataStreamReaderWriterSuite.scala
Loading
Please register or sign in to comment