-
- Downloads
[SPARK-20030][SS] Event-time-based timeout for MapGroupsWithState
## What changes were proposed in this pull request? Adding event time based timeout. The user sets the timeout timestamp directly using `KeyedState.setTimeoutTimestamp`. The keys times out when the watermark crosses the timeout timestamp. ## How was this patch tested? Unit tests Author: Tathagata Das <tathagata.das1565@gmail.com> Closes #17361 from tdas/SPARK-20030.
Showing
- sql/catalyst/src/main/java/org/apache/spark/sql/streaming/KeyedStateTimeout.java 17 additions, 5 deletions...ava/org/apache/spark/sql/streaming/KeyedStateTimeout.java
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/UnsupportedOperationChecker.scala 58 additions, 38 deletions...k/sql/catalyst/analysis/UnsupportedOperationChecker.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/object.scala 2 additions, 1 deletion.../org/apache/spark/sql/catalyst/plans/logical/object.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/UnsupportedOperationsSuite.scala 16 additions, 0 deletions...rk/sql/catalyst/analysis/UnsupportedOperationsSuite.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala 1 addition, 2 deletions...cala/org/apache/spark/sql/execution/SparkStrategies.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FlatMapGroupsWithStateExec.scala 57 additions, 30 deletions.../sql/execution/streaming/FlatMapGroupsWithStateExec.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/IncrementalExecution.scala 4 additions, 1 deletion.../spark/sql/execution/streaming/IncrementalExecution.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/KeyedStateImpl.scala 103 additions, 36 deletions...apache/spark/sql/execution/streaming/KeyedStateImpl.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/statefulOperators.scala 6 additions, 8 deletions...che/spark/sql/execution/streaming/statefulOperators.scala
- sql/core/src/main/scala/org/apache/spark/sql/streaming/KeyedState.scala 84 additions, 13 deletions...ain/scala/org/apache/spark/sql/streaming/KeyedState.scala
- sql/core/src/test/scala/org/apache/spark/sql/streaming/FlatMapGroupsWithStateSuite.scala 268 additions, 134 deletions...che/spark/sql/streaming/FlatMapGroupsWithStateSuite.scala
Loading
Please register or sign in to comment