-
- Downloads
Minor cleanup of ShuffleMapStage.outputLocs code.
I was looking at this code and found the documentation to be insufficient. I added more documentation, and refactored some relevant code path slightly to improve encapsulation. There are more that I want to do, but I want to get these changes in before doing more work. My goal is to reduce exposing internal fields directly in ShuffleMapStage to improve encapsulation. After this change, DAGScheduler no longer directly writes outputLocs. There are still 3 places that reads outputLocs directly, but we can change those later. Author: Reynold Xin <rxin@databricks.com> Closes #9175 from rxin/stage-cleanup.
Showing
- core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala 9 additions, 19 deletions.../main/scala/org/apache/spark/scheduler/DAGScheduler.scala
- core/src/main/scala/org/apache/spark/scheduler/ResultStage.scala 5 additions, 0 deletions...c/main/scala/org/apache/spark/scheduler/ResultStage.scala
- core/src/main/scala/org/apache/spark/scheduler/ShuffleMapStage.scala 21 additions, 0 deletions...in/scala/org/apache/spark/scheduler/ShuffleMapStage.scala
- core/src/main/scala/org/apache/spark/scheduler/Stage.scala 4 additions, 1 deletioncore/src/main/scala/org/apache/spark/scheduler/Stage.scala
Loading
Please register or sign in to comment