-
- Downloads
[SPARK-4857] [CORE] Adds Executor membership events to SparkListener
Adds onExecutorAdded and onExecutorRemoved events to the SparkListener. This will allow a client to get notified when an executor has been added/removed and provide additional information such as how many vcores it is consuming. In addition, this commit adds a SparkListenerAdapter to the Java API that provides default implementations to the SparkListener. This is to get around the fact that default implementations for traits don't work in Java. Having Java clients extend SparkListenerAdapter moving forward will prevent breakage in java when we add new events to SparkListener. Author: Kostas Sakellis <kostas@cloudera.com> Closes #3711 from ksakellis/kostas-spark-4857 and squashes the following commits: 946d2c5 [Kostas Sakellis] Added executorAdded/Removed events to MesosSchedulerBackend b1d054a [Kostas Sakellis] Remove executorInfo from ExecutorRemoved event 1727b38 [Kostas Sakellis] Renamed ExecutorDetails back to ExecutorInfo and other CR feedback 14fe78d [Kostas Sakellis] Added executor added/removed events to json protocol 93d087b [Kostas Sakellis] [SPARK-4857] [CORE] Adds Executor membership events to SparkListener
Showing
- core/src/main/java/org/apache/spark/JavaSparkListener.java 97 additions, 0 deletionscore/src/main/java/org/apache/spark/JavaSparkListener.java
- core/src/main/scala/org/apache/spark/deploy/master/ApplicationInfo.scala 7 additions, 7 deletions...cala/org/apache/spark/deploy/master/ApplicationInfo.scala
- core/src/main/scala/org/apache/spark/deploy/master/ExecutorDesc.scala 2 additions, 2 deletions...n/scala/org/apache/spark/deploy/master/ExecutorDesc.scala
- core/src/main/scala/org/apache/spark/deploy/master/Master.scala 1 addition, 1 deletion...rc/main/scala/org/apache/spark/deploy/master/Master.scala
- core/src/main/scala/org/apache/spark/deploy/master/WorkerInfo.scala 3 additions, 3 deletions...ain/scala/org/apache/spark/deploy/master/WorkerInfo.scala
- core/src/main/scala/org/apache/spark/deploy/master/ui/ApplicationPage.scala 2 additions, 2 deletions...a/org/apache/spark/deploy/master/ui/ApplicationPage.scala
- core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala 4 additions, 0 deletions...ala/org/apache/spark/scheduler/EventLoggingListener.scala
- core/src/main/scala/org/apache/spark/scheduler/SparkListener.scala 21 additions, 1 deletion...main/scala/org/apache/spark/scheduler/SparkListener.scala
- core/src/main/scala/org/apache/spark/scheduler/SparkListenerBus.scala 4 additions, 0 deletions...n/scala/org/apache/spark/scheduler/SparkListenerBus.scala
- core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala 5 additions, 1 deletion...ark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
- core/src/main/scala/org/apache/spark/scheduler/cluster/ExecutorData.scala 3 additions, 3 deletions...ala/org/apache/spark/scheduler/cluster/ExecutorData.scala
- core/src/main/scala/org/apache/spark/scheduler/cluster/ExecutorInfo.scala 45 additions, 0 deletions...ala/org/apache/spark/scheduler/cluster/ExecutorInfo.scala
- core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackend.scala 25 additions, 7 deletions...spark/scheduler/cluster/mesos/MesosSchedulerBackend.scala
- core/src/main/scala/org/apache/spark/util/JsonProtocol.scala 39 additions, 1 deletioncore/src/main/scala/org/apache/spark/util/JsonProtocol.scala
- core/src/test/scala/org/apache/spark/scheduler/EventLoggingListenerSuite.scala 2 additions, 1 deletion...rg/apache/spark/scheduler/EventLoggingListenerSuite.scala
- core/src/test/scala/org/apache/spark/scheduler/SparkListenerWithClusterSuite.scala 62 additions, 0 deletions...pache/spark/scheduler/SparkListenerWithClusterSuite.scala
- core/src/test/scala/org/apache/spark/scheduler/mesos/MesosSchedulerBackendSuite.scala 12 additions, 4 deletions...he/spark/scheduler/mesos/MesosSchedulerBackendSuite.scala
- core/src/test/scala/org/apache/spark/util/JsonProtocolSuite.scala 41 additions, 0 deletions.../test/scala/org/apache/spark/util/JsonProtocolSuite.scala
Loading
Please register or sign in to comment