-
- Downloads
[SPARK-12244][SPARK-12245][STREAMING] Rename trackStateByKey to mapWithState...
[SPARK-12244][SPARK-12245][STREAMING] Rename trackStateByKey to mapWithState and change tracking function signature SPARK-12244: Based on feedback from early users and personal experience attempting to explain it, the name trackStateByKey had two problem. "trackState" is a completely new term which really does not give any intuition on what the operation is the resultant data stream of objects returned by the function is called in docs as the "emitted" data for the lack of a better. "mapWithState" makes sense because the API is like a mapping function like (Key, Value) => T with State as an additional parameter. The resultant data stream is "mapped data". So both problems are solved. SPARK-12245: From initial experiences, not having the key in the function makes it hard to return mapped stuff, as the whole information of the records is not there. Basically the user is restricted to doing something like mapValue() instead of map(). So adding the key as a parameter. Author: Tathagata Das <tathagata.das1565@gmail.com> Closes #10224 from tdas/rename.
Showing
- examples/src/main/java/org/apache/spark/examples/streaming/JavaStatefulNetworkWordCount.java 8 additions, 8 deletions...park/examples/streaming/JavaStatefulNetworkWordCount.java
- examples/src/main/scala/org/apache/spark/examples/streaming/StatefulNetworkWordCount.scala 6 additions, 6 deletions...e/spark/examples/streaming/StatefulNetworkWordCount.scala
- extras/java8-tests/src/test/java/org/apache/spark/streaming/Java8APISuite.java 8 additions, 10 deletions...c/test/java/org/apache/spark/streaming/Java8APISuite.java
- streaming/src/main/scala/org/apache/spark/streaming/State.scala 11 additions, 9 deletions...ing/src/main/scala/org/apache/spark/streaming/State.scala
- streaming/src/main/scala/org/apache/spark/streaming/StateSpec.scala 91 additions, 69 deletions...src/main/scala/org/apache/spark/streaming/StateSpec.scala
- streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaMapWithStateDStream.scala 10 additions, 10 deletions...he/spark/streaming/api/java/JavaMapWithStateDStream.scala
- streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala 22 additions, 28 deletions...org/apache/spark/streaming/api/java/JavaPairDStream.scala
- streaming/src/main/scala/org/apache/spark/streaming/dstream/MapWithStateDStream.scala 30 additions, 31 deletions.../apache/spark/streaming/dstream/MapWithStateDStream.scala
- streaming/src/main/scala/org/apache/spark/streaming/dstream/PairDStreamFunctions.scala 19 additions, 22 deletions...apache/spark/streaming/dstream/PairDStreamFunctions.scala
- streaming/src/main/scala/org/apache/spark/streaming/rdd/MapWithStateRDD.scala 47 additions, 52 deletions...cala/org/apache/spark/streaming/rdd/MapWithStateRDD.scala
- streaming/src/test/java/org/apache/spark/streaming/JavaMapWithStateSuite.java 24 additions, 24 deletions...ava/org/apache/spark/streaming/JavaMapWithStateSuite.java
- streaming/src/test/scala/org/apache/spark/streaming/MapWithStateSuite.scala 56 additions, 56 deletions.../scala/org/apache/spark/streaming/MapWithStateSuite.scala
- streaming/src/test/scala/org/apache/spark/streaming/rdd/MapWithStateRDDSuite.scala 57 additions, 57 deletions...org/apache/spark/streaming/rdd/MapWithStateRDDSuite.scala
Loading
Please register or sign in to comment