-
- Downloads
[SPARK-7139] [STREAMING] Allow received block metadata to be saved to WAL and...
[SPARK-7139] [STREAMING] Allow received block metadata to be saved to WAL and recovered on driver failure - Enabled ReceivedBlockTracker WAL by default - Stored block metadata in the WAL - Optimized WALBackedBlockRDD by skipping block fetch when the block is known to not exist in Spark Author: Tathagata Das <tathagata.das1565@gmail.com> Closes #5732 from tdas/SPARK-7139 and squashes the following commits: 575476e [Tathagata Das] Added more tests to get 100% coverage of the WALBackedBlockRDD 19668ba [Tathagata Das] Merge remote-tracking branch 'apache-github/master' into SPARK-7139 685fab3 [Tathagata Das] Addressed comments in PR 637bc9c [Tathagata Das] Changed segment to handle 466212c [Tathagata Das] Merge remote-tracking branch 'apache-github/master' into SPARK-7139 5f67a59 [Tathagata Das] Fixed HdfsUtils to handle append in local file system 1bc5bc3 [Tathagata Das] Fixed bug on unexpected recovery d06fa21 [Tathagata Das] Enabled ReceivedBlockTracker by default, stored block metadata and optimized block fetching in WALBackedBlockRDD
Showing
- core/src/main/scala/org/apache/spark/rdd/BlockRDD.scala 4 additions, 4 deletionscore/src/main/scala/org/apache/spark/rdd/BlockRDD.scala
- streaming/src/main/scala/org/apache/spark/streaming/dstream/ReceiverInputDStream.scala 24 additions, 25 deletions...apache/spark/streaming/dstream/ReceiverInputDStream.scala
- streaming/src/main/scala/org/apache/spark/streaming/rdd/WriteAheadLogBackedBlockRDD.scala 98 additions, 58 deletions...che/spark/streaming/rdd/WriteAheadLogBackedBlockRDD.scala
- streaming/src/main/scala/org/apache/spark/streaming/receiver/ReceiverSupervisorImpl.scala 1 addition, 1 deletion...che/spark/streaming/receiver/ReceiverSupervisorImpl.scala
- streaming/src/main/scala/org/apache/spark/streaming/scheduler/ReceivedBlockInfo.scala 28 additions, 2 deletions.../apache/spark/streaming/scheduler/ReceivedBlockInfo.scala
- streaming/src/main/scala/org/apache/spark/streaming/scheduler/ReceivedBlockTracker.scala 9 additions, 15 deletions...ache/spark/streaming/scheduler/ReceivedBlockTracker.scala
- streaming/src/main/scala/org/apache/spark/streaming/scheduler/ReceiverTracker.scala 1 addition, 0 deletions...rg/apache/spark/streaming/scheduler/ReceiverTracker.scala
- streaming/src/main/scala/org/apache/spark/streaming/util/HdfsUtils.scala 1 addition, 1 deletion...ain/scala/org/apache/spark/streaming/util/HdfsUtils.scala
- streaming/src/test/scala/org/apache/spark/streaming/ReceivedBlockTrackerSuite.scala 30 additions, 25 deletions...rg/apache/spark/streaming/ReceivedBlockTrackerSuite.scala
- streaming/src/test/scala/org/apache/spark/streaming/rdd/WriteAheadLogBackedBlockRDDSuite.scala 85 additions, 22 deletions...park/streaming/rdd/WriteAheadLogBackedBlockRDDSuite.scala
Loading
Please register or sign in to comment