-
- Downloads
[SPARK-14176][SQL] Add DataFrameWriter.trigger to set the stream batch period
## What changes were proposed in this pull request? Add a processing time trigger to control the batch processing speed ## How was this patch tested? Unit tests Author: Shixiong Zhu <shixiong@databricks.com> Closes #11976 from zsxwing/trigger.
Showing
- sql/core/src/main/scala/org/apache/spark/sql/ContinuousQueryManager.scala 9 additions, 2 deletions...n/scala/org/apache/spark/sql/ContinuousQueryManager.scala
- sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala 33 additions, 1 deletion...src/main/scala/org/apache/spark/sql/DataFrameWriter.scala
- sql/core/src/main/scala/org/apache/spark/sql/Trigger.scala 133 additions, 0 deletionssql/core/src/main/scala/org/apache/spark/sql/Trigger.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala 15 additions, 9 deletions...pache/spark/sql/execution/streaming/StreamExecution.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/TriggerExecutor.scala 72 additions, 0 deletions...pache/spark/sql/execution/streaming/TriggerExecutor.scala
- sql/core/src/test/scala/org/apache/spark/sql/ProcessingTimeSuite.scala 40 additions, 0 deletions...test/scala/org/apache/spark/sql/ProcessingTimeSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/StreamTest.scala 5 additions, 1 deletion...core/src/test/scala/org/apache/spark/sql/StreamTest.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/ProcessingTimeExecutorSuite.scala 78 additions, 0 deletions...sql/execution/streaming/ProcessingTimeExecutorSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/streaming/DataFrameReaderWriterSuite.scala 28 additions, 0 deletions...ache/spark/sql/streaming/DataFrameReaderWriterSuite.scala
Loading
Please register or sign in to comment