Skip to content
Snippets Groups Projects
Commit de700d31 authored by Reynold Xin's avatar Reynold Xin
Browse files

[SPARK-3709] Executors don't always report broadcast block removal properly back to the driver

The problem was that the 2nd argument in RemoveBroadcast is not tellMaster! It is "removeFromDriver". Basically when removeFromDriver is not true, we don't report broadcast block removal back to the driver, and then other executors mistakenly think that the executor would still have the block, and try to fetch from it.

cc @tdas

Author: Reynold Xin <rxin@apache.org>

Closes #2588 from rxin/debug and squashes the following commits:

6dab2e3 [Reynold Xin] Don't log random messages.
f430686 [Reynold Xin] Always report broadcast removal back to master.
2a13f70 [Reynold Xin] iii
parent 6b79bfb4
No related branches found
No related tags found
No related merge requests found
......@@ -200,6 +200,6 @@ final class NioBlockTransferService(conf: SparkConf, securityManager: SecurityMa
val buffer = blockDataManager.getBlockData(blockId).orNull
logDebug("GetBlock " + blockId + " used " + Utils.getUsedTimeMs(startTimeMs)
+ " and got buffer " + buffer)
buffer.nioByteBuffer()
if (buffer == null) null else buffer.nioByteBuffer()
}
}
......@@ -58,9 +58,9 @@ class BlockManagerSlaveActor(
SparkEnv.get.shuffleManager.unregisterShuffle(shuffleId)
}
case RemoveBroadcast(broadcastId, tellMaster) =>
case RemoveBroadcast(broadcastId, _) =>
doAsync[Int]("removing broadcast " + broadcastId, sender) {
blockManager.removeBroadcast(broadcastId, tellMaster)
blockManager.removeBroadcast(broadcastId, tellMaster = true)
}
case GetBlockStatus(blockId, _) =>
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment