-
- Downloads
[SPARK-15889][SQL][STREAMING] Add a unique id to ContinuousQuery
## What changes were proposed in this pull request? ContinuousQueries have names that are unique across all the active ones. However, when queries are rapidly restarted with same name, it causes races conditions with the listener. A listener event from a stopped query can arrive after the query has been restarted, leading to complexities in monitoring infrastructure. Along with this change, I have also consolidated all the messy code paths to start queries with different sinks. ## How was this patch tested? Added unit tests, and existing unit tests. Author: Tathagata Das <tathagata.das1565@gmail.com> Closes #13613 from tdas/SPARK-15889.
Showing
- sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala 17 additions, 52 deletions...src/main/scala/org/apache/spark/sql/DataFrameWriter.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala 5 additions, 3 deletions...pache/spark/sql/execution/streaming/StreamExecution.scala
- sql/core/src/main/scala/org/apache/spark/sql/streaming/ContinuousQuery.scala 10 additions, 1 deletion...cala/org/apache/spark/sql/streaming/ContinuousQuery.scala
- sql/core/src/main/scala/org/apache/spark/sql/streaming/ContinuousQueryInfo.scala 4 additions, 1 deletion.../org/apache/spark/sql/streaming/ContinuousQueryInfo.scala
- sql/core/src/main/scala/org/apache/spark/sql/streaming/ContinuousQueryManager.scala 61 additions, 13 deletions...g/apache/spark/sql/streaming/ContinuousQueryManager.scala
- sql/core/src/test/scala/org/apache/spark/sql/streaming/ContinuousQueryListenerSuite.scala 14 additions, 3 deletions...he/spark/sql/streaming/ContinuousQueryListenerSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/streaming/ContinuousQueryManagerSuite.scala 8 additions, 21 deletions...che/spark/sql/streaming/ContinuousQueryManagerSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/streaming/ContinuousQuerySuite.scala 42 additions, 1 deletion...org/apache/spark/sql/streaming/ContinuousQuerySuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamTest.scala 6 additions, 6 deletions...est/scala/org/apache/spark/sql/streaming/StreamTest.scala
Loading
Please register or sign in to comment