-
- Downloads
[SPARK-4028][Streaming] ReceivedBlockHandler interface to abstract the...
[SPARK-4028][Streaming] ReceivedBlockHandler interface to abstract the functionality of storage of received data As part of the initiative to prevent data loss on streaming driver failure, this JIRA tracks the subtask of implementing a ReceivedBlockHandler, that abstracts the functionality of storage of received data blocks. The default implementation will maintain the current behavior of storing the data into BlockManager. The optional implementation will store the data to both BlockManager as well as a write ahead log. Author: Tathagata Das <tathagata.das1565@gmail.com> Closes #2940 from tdas/driver-ha-rbh and squashes the following commits: 78a4aaa [Tathagata Das] Fixed bug causing test failures. f192f47 [Tathagata Das] Fixed import order. df5f320 [Tathagata Das] Updated code to use ReceivedBlockStoreResult as the return type for handler's storeBlock 33c30c9 [Tathagata Das] Added license, and organized imports. 2f025b3 [Tathagata Das] Updates based on PR comments. 18aec1e [Tathagata Das] Moved ReceivedBlockInfo back into spark.streaming.scheduler package 95a4987 [Tathagata Das] Added ReceivedBlockHandler and its associated tests
Showing
- streaming/src/main/scala/org/apache/spark/streaming/dstream/ReceiverInputDStream.scala 4 additions, 3 deletions...apache/spark/streaming/dstream/ReceiverInputDStream.scala
- streaming/src/main/scala/org/apache/spark/streaming/receiver/ReceivedBlock.scala 35 additions, 0 deletions...a/org/apache/spark/streaming/receiver/ReceivedBlock.scala
- streaming/src/main/scala/org/apache/spark/streaming/receiver/ReceivedBlockHandler.scala 193 additions, 0 deletions...pache/spark/streaming/receiver/ReceivedBlockHandler.scala
- streaming/src/main/scala/org/apache/spark/streaming/receiver/ReceiverSupervisorImpl.scala 51 additions, 37 deletions...che/spark/streaming/receiver/ReceiverSupervisorImpl.scala
- streaming/src/main/scala/org/apache/spark/streaming/scheduler/BatchInfo.scala 1 addition, 1 deletion...cala/org/apache/spark/streaming/scheduler/BatchInfo.scala
- streaming/src/main/scala/org/apache/spark/streaming/scheduler/JobSet.scala 2 additions, 1 deletion...n/scala/org/apache/spark/streaming/scheduler/JobSet.scala
- streaming/src/main/scala/org/apache/spark/streaming/scheduler/ReceivedBlockInfo.scala 28 additions, 0 deletions.../apache/spark/streaming/scheduler/ReceivedBlockInfo.scala
- streaming/src/main/scala/org/apache/spark/streaming/scheduler/ReceiverTracker.scala 10 additions, 14 deletions...rg/apache/spark/streaming/scheduler/ReceiverTracker.scala
- streaming/src/main/scala/org/apache/spark/streaming/util/WriteAheadLogRandomReader.scala 0 additions, 1 deletion...ache/spark/streaming/util/WriteAheadLogRandomReader.scala
- streaming/src/test/scala/org/apache/spark/streaming/ReceivedBlockHandlerSuite.scala 258 additions, 0 deletions...rg/apache/spark/streaming/ReceivedBlockHandlerSuite.scala
- streaming/src/test/scala/org/apache/spark/streaming/util/WriteAheadLogSuite.scala 21 additions, 13 deletions.../org/apache/spark/streaming/util/WriteAheadLogSuite.scala
Loading
Please register or sign in to comment