-
- Downloads
[SPARK-8975] [STREAMING] Adds a mechanism to send a new rate from the driver to the block generator
First step for [SPARK-7398](https://issues.apache.org/jira/browse/SPARK-7398). tdas huitseeker Author: Iulian Dragos <jaguarul@gmail.com> Author: François Garillot <francois@garillot.net> Closes #7471 from dragos/topic/streaming-bp/dynamic-rate and squashes the following commits: 8941cf9 [Iulian Dragos] Renames and other nitpicks. 162d9e5 [Iulian Dragos] Use Reflection for accessing truly private `executor` method and use the listener bus to know when receivers have registered (`onStart` is called before receivers have registered, leading to flaky behavior). 210f495 [Iulian Dragos] Revert "Added a few tests that measure the receiver’s rate." 0c51959 [Iulian Dragos] Added a few tests that measure the receiver’s rate. 261a051 [Iulian Dragos] - removed field to hold the current rate limit in rate limiter - made rate limit a Long and default to Long.MaxValue (consequence of the above) - removed custom `waitUntil` and replaced it by `eventually` cd1397d [Iulian Dragos] Add a test for the propagation of a new rate limit from driver to receivers. 6369b30 [Iulian Dragos] Merge pull request #15 from huitseeker/SPARK-8975 d15de42 [François Garillot] [SPARK-8975][Streaming] Adds Ratelimiter unit tests w.r.t. spark.streaming.receiver.maxRate 4721c7d [François Garillot] [SPARK-8975][Streaming] Add a mechanism to send a new rate from the driver to the block generator
Showing
- streaming/src/main/scala/org/apache/spark/streaming/receiver/RateLimiter.scala 25 additions, 5 deletions...ala/org/apache/spark/streaming/receiver/RateLimiter.scala
- streaming/src/main/scala/org/apache/spark/streaming/receiver/Receiver.scala 1 addition, 1 deletion.../scala/org/apache/spark/streaming/receiver/Receiver.scala
- streaming/src/main/scala/org/apache/spark/streaming/receiver/ReceiverMessage.scala 2 additions, 1 deletion...org/apache/spark/streaming/receiver/ReceiverMessage.scala
- streaming/src/main/scala/org/apache/spark/streaming/receiver/ReceiverSupervisor.scala 3 additions, 0 deletions.../apache/spark/streaming/receiver/ReceiverSupervisor.scala
- streaming/src/main/scala/org/apache/spark/streaming/receiver/ReceiverSupervisorImpl.scala 6 additions, 0 deletions...che/spark/streaming/receiver/ReceiverSupervisorImpl.scala
- streaming/src/main/scala/org/apache/spark/streaming/scheduler/ReceiverTracker.scala 8 additions, 1 deletion...rg/apache/spark/streaming/scheduler/ReceiverTracker.scala
- streaming/src/test/scala/org/apache/spark/streaming/receiver/RateLimiterSuite.scala 46 additions, 0 deletions...rg/apache/spark/streaming/receiver/RateLimiterSuite.scala
- streaming/src/test/scala/org/apache/spark/streaming/scheduler/ReceiverTrackerSuite.scala 62 additions, 0 deletions...ache/spark/streaming/scheduler/ReceiverTrackerSuite.scala
Loading
Please register or sign in to comment