Skip to content
Snippets Groups Projects
Commit 7fa960e6 authored by Davies Liu's avatar Davies Liu Committed by Josh Rosen
Browse files

[SPARK-5363] Fix bug in PythonRDD: remove() inside iterator is not safe

Removing elements from a mutable HashSet while iterating over it can cause the
iteration to incorrectly skip over entries that were not removed. If this
happened, PythonRDD would write fewer broadcast variables than the Python
worker was expecting to read, which would cause the Python worker to hang
indefinitely.

Author: Davies Liu <davies@databricks.com>

Closes #4776 from davies/fix_hang and squashes the following commits:

a4384a5 [Davies Liu] fix bug: remvoe() inside iterator is not safe
parent cfff397f
No related branches found
No related tags found
No related merge requests found
...@@ -219,14 +219,13 @@ private[spark] class PythonRDD( ...@@ -219,14 +219,13 @@ private[spark] class PythonRDD(
val oldBids = PythonRDD.getWorkerBroadcasts(worker) val oldBids = PythonRDD.getWorkerBroadcasts(worker)
val newBids = broadcastVars.map(_.id).toSet val newBids = broadcastVars.map(_.id).toSet
// number of different broadcasts // number of different broadcasts
val cnt = oldBids.diff(newBids).size + newBids.diff(oldBids).size val toRemove = oldBids.diff(newBids)
val cnt = toRemove.size + newBids.diff(oldBids).size
dataOut.writeInt(cnt) dataOut.writeInt(cnt)
for (bid <- oldBids) { for (bid <- toRemove) {
if (!newBids.contains(bid)) { // remove the broadcast from worker
// remove the broadcast from worker dataOut.writeLong(- bid - 1) // bid >= 0
dataOut.writeLong(- bid - 1) // bid >= 0 oldBids.remove(bid)
oldBids.remove(bid)
}
} }
for (broadcast <- broadcastVars) { for (broadcast <- broadcastVars) {
if (!oldBids.contains(broadcast.id)) { if (!oldBids.contains(broadcast.id)) {
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment