Skip to content
Snippets Groups Projects
Commit 09deb3ee authored by Andrew Or's avatar Andrew Or Committed by Patrick Wendell
Browse files

[SPARK-2144] ExecutorsPage reports incorrect # of RDD blocks

This is reproducible whenever we drop a block because of memory pressure.

This is because StorageStatusListener actually never removes anything from the block maps of its StorageStatuses. Instead, when a block is dropped, it sets the block's storage level to `StorageLevel.NONE`, when it should just remove it from the map.

This PR includes this simple fix.

Author: Andrew Or <andrewor14@gmail.com>

Closes #1080 from andrewor14/ui-blocks and squashes the following commits:

fcf9f1a [Andrew Or] Remove BlockStatus if it is no longer cached
parent 23a12ce2
No related branches found
No related tags found
No related merge requests found
......@@ -37,7 +37,11 @@ class StorageStatusListener extends SparkListener {
val filteredStatus = storageStatusList.find(_.blockManagerId.executorId == execId)
filteredStatus.foreach { storageStatus =>
updatedBlocks.foreach { case (blockId, updatedStatus) =>
storageStatus.blocks(blockId) = updatedStatus
if (updatedStatus.storageLevel == StorageLevel.NONE) {
storageStatus.blocks.remove(blockId)
} else {
storageStatus.blocks(blockId) = updatedStatus
}
}
}
}
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment