-
- Downloads
[SPARK-14719] WriteAheadLogBasedBlockHandler should ignore BlockManager put errors
WriteAheadLogBasedBlockHandler will currently throw exceptions if its BlockManager `put()` calls fail, even though those calls are only performed as a performance optimization. Instead, it should log and ignore exceptions during that `put()`. This is a longstanding issue that was masked by an incorrect test case. I think that we haven't noticed this in production because 1. most people probably use a `MEMORY_AND_DISK` storage level, and 2. typically, individual blocks may be small enough relative to the total storage memory such that they're able to evict blocks from previous batches, so `put()` failures here may be rare in practice. This patch fixes the faulty test and fixes the bug. /cc tdas Author: Josh Rosen <joshrosen@databricks.com> Closes #12484 from JoshRosen/received-block-hadndler-fix.
Showing
- streaming/src/main/scala/org/apache/spark/streaming/receiver/ReceivedBlockHandler.scala 14 additions, 8 deletions...pache/spark/streaming/receiver/ReceivedBlockHandler.scala
- streaming/src/test/scala/org/apache/spark/streaming/ReceivedBlockHandlerSuite.scala 26 additions, 23 deletions...rg/apache/spark/streaming/ReceivedBlockHandlerSuite.scala
Loading
Please register or sign in to comment