-
- Downloads
[SPARK-2321] Several progress API improvements / refactorings
This PR refactors / extends the status API introduced in #2696. - Change StatusAPI from a mixin trait to a class. Before, the new status API methods were directly accessible through SparkContext, whereas now they're accessed through a `sc.statusAPI` field. As long as we were going to add these methods directly to SparkContext, the mixin trait seemed like a good idea, but this might be simpler to reason about and may avoid pitfalls that I've run into while attempting to refactor other parts of SparkContext to use mixins (see #3071, for example). - Change the name from SparkStatusAPI to SparkStatusTracker. - Make `getJobIdsForGroup(null)` return ids for jobs that aren't associated with any job group. - Add `getActiveStageIds()` and `getActiveJobIds()` methods that return the ids of whatever's currently active in this SparkContext. This should simplify davies's progress bar code. Author: Josh Rosen <joshrosen@databricks.com> Closes #3197 from JoshRosen/progress-api-improvements and squashes the following commits: 30b0afa [Josh Rosen] Rename SparkStatusAPI to SparkStatusTracker. d1b08d8 [Josh Rosen] Add missing newlines 2cc7353 [Josh Rosen] Add missing file. d5eab1f [Josh Rosen] Add getActive[Stage|Job]Ids() methods. a227984 [Josh Rosen] getJobIdsForGroup(null) should return jobs for default group c47e294 [Josh Rosen] Remove StatusAPI mixin trait.
Showing
- core/src/main/scala/org/apache/spark/SparkContext.scala 67 additions, 1 deletioncore/src/main/scala/org/apache/spark/SparkContext.scala
- core/src/main/scala/org/apache/spark/SparkStatusTracker.scala 107 additions, 0 deletions.../src/main/scala/org/apache/spark/SparkStatusTracker.scala
- core/src/main/scala/org/apache/spark/api/java/JavaSparkContext.scala 2 additions, 19 deletions...in/scala/org/apache/spark/api/java/JavaSparkContext.scala
- core/src/main/scala/org/apache/spark/api/java/JavaSparkStatusTracker.scala 72 additions, 0 deletions...la/org/apache/spark/api/java/JavaSparkStatusTracker.scala
- core/src/test/scala/org/apache/spark/StatusTrackerSuite.scala 18 additions, 7 deletions.../src/test/scala/org/apache/spark/StatusTrackerSuite.scala
- examples/src/main/java/org/apache/spark/examples/JavaStatusTrackerDemo.java 3 additions, 3 deletions...java/org/apache/spark/examples/JavaStatusTrackerDemo.java
Loading
Please register or sign in to comment