-
- Downloads
[SPARK-18406][CORE] Race between end-of-task and completion iterator read lock release
## What changes were proposed in this pull request? When a TaskContext is not propagated properly to all child threads for the task, just like the reported cases in this issue, we fail to get to TID from TaskContext and that causes unable to release the lock and assertion failures. To resolve this, we have to explicitly pass the TID value to the `unlock` method. ## How was this patch tested? Add new failing regression test case in `RDDSuite`. Author: Xingbo Jiang <xingbo.jiang@databricks.com> Closes #18076 from jiangxb1987/completion-iterator.
Showing
- core/src/main/scala/org/apache/spark/network/BlockDataManager.scala 1 addition, 1 deletion...ain/scala/org/apache/spark/network/BlockDataManager.scala
- core/src/main/scala/org/apache/spark/storage/BlockInfoManager.scala 10 additions, 5 deletions...ain/scala/org/apache/spark/storage/BlockInfoManager.scala
- core/src/main/scala/org/apache/spark/storage/BlockManager.scala 21 additions, 8 deletions...rc/main/scala/org/apache/spark/storage/BlockManager.scala
- core/src/test/scala/org/apache/spark/rdd/RDDSuite.scala 17 additions, 1 deletioncore/src/test/scala/org/apache/spark/rdd/RDDSuite.scala
Loading
Please register or sign in to comment