-
- Downloads
[SPARK-14257][SQL] Allow multiple continuous queries to be started from the same DataFrame
## What changes were proposed in this pull request?
Make StreamingRelation store the closure to create the source in StreamExecution so that we can start multiple continuous queries from the same DataFrame.
## How was this patch tested?
`test("DataFrame reuse")`
Author: Shixiong Zhu <shixiong@databricks.com>
Closes #12049 from zsxwing/df-reuse.
Showing
- sql/core/src/main/scala/org/apache/spark/sql/ContinuousQueryManager.scala 10 additions, 2 deletions...n/scala/org/apache/spark/sql/ContinuousQueryManager.scala
- sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala 1 addition, 1 deletion...src/main/scala/org/apache/spark/sql/DataFrameReader.scala
- sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala 4 additions, 2 deletionssql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala 4 additions, 4 deletions...pache/spark/sql/execution/streaming/StreamExecution.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamingRelation.scala 24 additions, 3 deletions...che/spark/sql/execution/streaming/StreamingRelation.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/memory.scala 2 additions, 4 deletions...ala/org/apache/spark/sql/execution/streaming/memory.scala
- sql/core/src/test/scala/org/apache/spark/sql/StreamTest.scala 3 additions, 2 deletions...core/src/test/scala/org/apache/spark/sql/StreamTest.scala
- sql/core/src/test/scala/org/apache/spark/sql/streaming/ContinuousQueryManagerSuite.scala 3 additions, 3 deletions...che/spark/sql/streaming/ContinuousQueryManagerSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/streaming/FileStreamSourceSuite.scala 6 additions, 4 deletions...rg/apache/spark/sql/streaming/FileStreamSourceSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamSuite.scala 61 additions, 1 deletion...st/scala/org/apache/spark/sql/streaming/StreamSuite.scala
Loading
Please register or sign in to comment