Skip to content
Snippets Groups Projects
Commit 642b246b authored by Zhang, Liye's avatar Zhang, Liye Committed by Andrew Or
Browse files

[SPARK-3941][CORE] _remainingmem should not increase twice when updateBlockInfo

In BlockManagermasterActor, _remainingMem would increase memSize for twice when updateBlockInfo if new storageLevel is invalid and old storageLevel is "useMemory". Also, _remainingMem should increase with original memory size instead of new memSize.

Author: Zhang, Liye <liye.zhang@intel.com>

Closes #2792 from liyezhang556520/spark-3941-remainMem and squashes the following commits:

3d487cc [Zhang, Liye] make the code concise
0380a32 [Zhang, Liye] [SPARK-3941][CORE] _remainingmem should not increase twice when updateBlockInfo
parent be2ec4a9
No related branches found
No related tags found
No related merge requests found
...@@ -457,16 +457,18 @@ private[spark] class BlockManagerInfo( ...@@ -457,16 +457,18 @@ private[spark] class BlockManagerInfo(
if (_blocks.containsKey(blockId)) { if (_blocks.containsKey(blockId)) {
// The block exists on the slave already. // The block exists on the slave already.
val originalLevel: StorageLevel = _blocks.get(blockId).storageLevel val blockStatus: BlockStatus = _blocks.get(blockId)
val originalLevel: StorageLevel = blockStatus.storageLevel
val originalMemSize: Long = blockStatus.memSize
if (originalLevel.useMemory) { if (originalLevel.useMemory) {
_remainingMem += memSize _remainingMem += originalMemSize
} }
} }
if (storageLevel.isValid) { if (storageLevel.isValid) {
/* isValid means it is either stored in-memory, on-disk or on-Tachyon. /* isValid means it is either stored in-memory, on-disk or on-Tachyon.
* But the memSize here indicates the data size in or dropped from memory, * The memSize here indicates the data size in or dropped from memory,
* tachyonSize here indicates the data size in or dropped from Tachyon, * tachyonSize here indicates the data size in or dropped from Tachyon,
* and the diskSize here indicates the data size in or dropped to disk. * and the diskSize here indicates the data size in or dropped to disk.
* They can be both larger than 0, when a block is dropped from memory to disk. * They can be both larger than 0, when a block is dropped from memory to disk.
...@@ -493,7 +495,6 @@ private[spark] class BlockManagerInfo( ...@@ -493,7 +495,6 @@ private[spark] class BlockManagerInfo(
val blockStatus: BlockStatus = _blocks.get(blockId) val blockStatus: BlockStatus = _blocks.get(blockId)
_blocks.remove(blockId) _blocks.remove(blockId)
if (blockStatus.storageLevel.useMemory) { if (blockStatus.storageLevel.useMemory) {
_remainingMem += blockStatus.memSize
logInfo("Removed %s on %s in memory (size: %s, free: %s)".format( logInfo("Removed %s on %s in memory (size: %s, free: %s)".format(
blockId, blockManagerId.hostPort, Utils.bytesToString(blockStatus.memSize), blockId, blockManagerId.hostPort, Utils.bytesToString(blockStatus.memSize),
Utils.bytesToString(_remainingMem))) Utils.bytesToString(_remainingMem)))
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment