-
- Downloads
[SPARK-19876][SS][WIP] OneTime Trigger Executor
## What changes were proposed in this pull request? An additional trigger and trigger executor that will execute a single trigger only. One can use this OneTime trigger to have more control over the scheduling of triggers. In addition, this patch requires an optimization to StreamExecution that logs a commit record at the end of successfully processing a batch. This new commit log will be used to determine the next batch (offsets) to process after a restart, instead of using the offset log itself to determine what batch to process next after restart; using the offset log to determine this would process the previously logged batch, always, thus not permitting a OneTime trigger feature. ## How was this patch tested? A number of existing tests have been revised. These tests all assumed that when restarting a stream, the last batch in the offset log is to be re-processed. Given that we now have a commit log that will tell us if that last batch was processed successfully, the results/assumptions of those tests needed to be revised accordingly. In addition, a OneTime trigger test was added to StreamingQuerySuite, which tests: - The semantics of OneTime trigger (i.e., on start, execute a single batch, then stop). - The case when the commit log was not able to successfully log the completion of a batch before restart, which would mean that we should fall back to what's in the offset log. - A OneTime trigger execution that results in an exception being thrown. marmbrus tdas zsxwing Please review http://spark.apache.org/contributing.html before opening a pull request. Author: Tyson Condie <tcondie@gmail.com> Author: Tathagata Das <tathagata.das1565@gmail.com> Closes #17219 from tcondie/stream-commit.
Showing
- external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaSourceSuite.scala 0 additions, 2 deletions...cala/org/apache/spark/sql/kafka010/KafkaSourceSuite.scala
- project/MimaExcludes.scala 5 additions, 1 deletionproject/MimaExcludes.scala
- python/pyspark/sql/streaming.py 17 additions, 46 deletionspython/pyspark/sql/streaming.py
- python/pyspark/sql/tests.py 15 additions, 2 deletionspython/pyspark/sql/tests.py
- sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/BatchCommitLog.scala 77 additions, 0 deletions...apache/spark/sql/execution/streaming/BatchCommitLog.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala 67 additions, 14 deletions...pache/spark/sql/execution/streaming/StreamExecution.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/TriggerExecutor.scala 11 additions, 0 deletions...pache/spark/sql/execution/streaming/TriggerExecutor.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/Triggers.scala 29 additions, 0 deletions...a/org/apache/spark/sql/execution/streaming/Triggers.scala
- sql/core/src/main/scala/org/apache/spark/sql/streaming/DataStreamWriter.scala 1 addition, 1 deletion...ala/org/apache/spark/sql/streaming/DataStreamWriter.scala
- sql/core/src/main/scala/org/apache/spark/sql/streaming/ProcessingTime.scala 18 additions, 18 deletions...scala/org/apache/spark/sql/streaming/ProcessingTime.scala
- sql/core/src/main/scala/org/apache/spark/sql/streaming/Trigger.java 105 additions, 0 deletions...rc/main/scala/org/apache/spark/sql/streaming/Trigger.java
- sql/core/src/test/scala/org/apache/spark/sql/streaming/EventTimeWatermarkSuite.scala 3 additions, 1 deletion.../apache/spark/sql/streaming/EventTimeWatermarkSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/streaming/FlatMapGroupsWithStateSuite.scala 2 additions, 1 deletion...che/spark/sql/streaming/FlatMapGroupsWithStateSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamSuite.scala 17 additions, 3 deletions...st/scala/org/apache/spark/sql/streaming/StreamSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamTest.scala 1 addition, 1 deletion...est/scala/org/apache/spark/sql/streaming/StreamTest.scala
- sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingAggregationSuite.scala 4 additions, 0 deletions...pache/spark/sql/streaming/StreamingAggregationSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQueryListenerSuite.scala 17 additions, 1 deletion...che/spark/sql/streaming/StreamingQueryListenerSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQuerySuite.scala 47 additions, 1 deletion.../org/apache/spark/sql/streaming/StreamingQuerySuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/streaming/test/DataStreamReaderWriterSuite.scala 3 additions, 2 deletions...park/sql/streaming/test/DataStreamReaderWriterSuite.scala
Loading
Please register or sign in to comment