-
- Downloads
[SPARK-18811] StreamSource resolution should happen in stream execution thread
## What changes were proposed in this pull request? When you start a stream, if we are trying to resolve the source of the stream, for example if we need to resolve partition columns, this could take a long time. This long execution time should not block the main thread where `query.start()` was called on. It should happen in the stream execution thread possibly before starting any triggers. ## How was this patch tested? Unit test added. Made sure test fails with no code changes. Author: Burak Yavuz <brkyvz@gmail.com> Closes #16238 from brkyvz/SPARK-18811.
Showing
- sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala 21 additions, 3 deletions...pache/spark/sql/execution/streaming/StreamExecution.scala
- sql/core/src/main/scala/org/apache/spark/sql/streaming/StreamingQueryManager.scala 1 addition, 13 deletions...rg/apache/spark/sql/streaming/StreamingQueryManager.scala
- sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQueryManagerSuite.scala 28 additions, 0 deletions...ache/spark/sql/streaming/StreamingQueryManagerSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/streaming/util/DefaultSource.scala 66 additions, 0 deletions...a/org/apache/spark/sql/streaming/util/DefaultSource.scala
Loading
Please register or sign in to comment