![hushan[胡珊]'s avatar](https://secure.gravatar.com/avatar/a8c5aaf1a1e1b61540e1f0665709db56527879b04a91da17b15fbf318b9c567c?s=48&d=identicon "hushan[胡珊]")
-
- Downloads
[SPARK-5259] [CORE] don't submit stage until its dependencies map outputs are registered
Track pending tasks by partition ID instead of Task objects. Before this change, failure & retry could result in a case where a stage got submitted before the map output from its dependencies get registered. This was due to an error in the condition for registering map outputs. Author: hushan[胡珊] <hushan@xiaomi.com> Author: Imran Rashid <irashid@cloudera.com> Closes #7699 from squito/SPARK-5259.
Showing
- core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala 6 additions, 6 deletions.../main/scala/org/apache/spark/scheduler/DAGScheduler.scala
- core/src/main/scala/org/apache/spark/scheduler/Stage.scala 1 addition, 1 deletioncore/src/main/scala/org/apache/spark/scheduler/Stage.scala
- core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala 2 additions, 2 deletions...ain/scala/org/apache/spark/scheduler/TaskSetManager.scala
- core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala 182 additions, 15 deletions.../scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
Loading
Please register or sign in to comment