-
- Downloads
[SPARK-3660][STREAMING] Initial RDD for updateStateByKey transformation
SPARK-3660 : Initial RDD for updateStateByKey transformation I have added a sample StatefulNetworkWordCountWithInitial inspired by StatefulNetworkWordCount. Please let me know if any changes are required. Author: Soumitra Kumar <kumar.soumitra@gmail.com> Closes #2665 from soumitrak/master and squashes the following commits: ee8980b [Soumitra Kumar] Fixed copy/paste issue. 304f636 [Soumitra Kumar] Added simpler version of updateStateByKey API with initialRDD and test. 9781135 [Soumitra Kumar] Fixed test, and renamed variable. 3da51a2 [Soumitra Kumar] Adding updateStateByKey with initialRDD API to JavaPairDStream. 2f78f7e [Soumitra Kumar] Merge remote-tracking branch 'upstream/master' d4fdd18 [Soumitra Kumar] Renamed variable and moved method. d0ce2cd [Soumitra Kumar] Merge remote-tracking branch 'upstream/master' 31399a4 [Soumitra Kumar] Merge remote-tracking branch 'upstream/master' 4efa58b [Soumitra Kumar] [SPARK-3660][STREAMING] Initial RDD for updateStateByKey transformation 8f40ca0 [Soumitra Kumar] Merge remote-tracking branch 'upstream/master' dde4271 [Soumitra Kumar] Merge remote-tracking branch 'upstream/master' fdd7db3 [Soumitra Kumar] Adding support of initial value for state update. SPARK-3660 : Initial RDD for updateStateByKey transformation
Showing
- examples/src/main/scala/org/apache/spark/examples/streaming/StatefulNetworkWordCount.scala 11 additions, 2 deletions...e/spark/examples/streaming/StatefulNetworkWordCount.scala
- streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala 19 additions, 0 deletions...org/apache/spark/streaming/api/java/JavaPairDStream.scala
- streaming/src/main/scala/org/apache/spark/streaming/dstream/PairDStreamFunctions.scala 48 additions, 1 deletion...apache/spark/streaming/dstream/PairDStreamFunctions.scala
- streaming/src/main/scala/org/apache/spark/streaming/dstream/StateDStream.scala 39 additions, 31 deletions...ala/org/apache/spark/streaming/dstream/StateDStream.scala
- streaming/src/test/java/org/apache/spark/streaming/JavaAPISuite.java 49 additions, 4 deletions...rc/test/java/org/apache/spark/streaming/JavaAPISuite.java
- streaming/src/test/scala/org/apache/spark/streaming/BasicOperationsSuite.scala 74 additions, 0 deletions...ala/org/apache/spark/streaming/BasicOperationsSuite.scala
Loading
Please register or sign in to comment