-
- Downloads
[SPARK-11141][STREAMING] Batch ReceivedBlockTrackerLogEvents for WAL writes
When using S3 as a directory for WALs, the writes take too long. The driver gets very easily bottlenecked when multiple receivers send AddBlock events to the ReceiverTracker. This PR adds batching of events in the ReceivedBlockTracker so that receivers don't get blocked by the driver for too long. cc zsxwing tdas Author: Burak Yavuz <brkyvz@gmail.com> Closes #9143 from brkyvz/batch-wal-writes.
Showing
- streaming/src/main/scala/org/apache/spark/streaming/scheduler/ReceivedBlockTracker.scala 41 additions, 21 deletions...ache/spark/streaming/scheduler/ReceivedBlockTracker.scala
- streaming/src/main/scala/org/apache/spark/streaming/scheduler/ReceiverTracker.scala 22 additions, 3 deletions...rg/apache/spark/streaming/scheduler/ReceiverTracker.scala
- streaming/src/main/scala/org/apache/spark/streaming/util/BatchedWriteAheadLog.scala 223 additions, 0 deletions...rg/apache/spark/streaming/util/BatchedWriteAheadLog.scala
- streaming/src/main/scala/org/apache/spark/streaming/util/WriteAheadLogUtils.scala 20 additions, 1 deletion.../org/apache/spark/streaming/util/WriteAheadLogUtils.scala
- streaming/src/test/scala/org/apache/spark/streaming/util/WriteAheadLogSuite.scala 339 additions, 167 deletions.../org/apache/spark/streaming/util/WriteAheadLogSuite.scala
- streaming/src/test/scala/org/apache/spark/streaming/util/WriteAheadLogUtilsSuite.scala 122 additions, 0 deletions...apache/spark/streaming/util/WriteAheadLogUtilsSuite.scala
Loading
Please register or sign in to comment