-
- Downloads
[SPARK-13001][CORE][MESOS] Prevent getting offers when reached max cores
Similar to https://github.com/apache/spark/pull/8639 This change rejects offers for 120s when reached `spark.cores.max` in coarse-grained mode to mitigate offer starvation. This prevents Mesos to send us offers again and again, starving other frameworks. This is especially problematic when running many small frameworks on the same Mesos cluster, e.g. many small Sparks streaming jobs, and cause the bigger spark jobs to stop receiving offers. By rejecting the offers for a long period of time, they become available to those other frameworks. Author: Sebastien Rainville <sebastien@hopper.com> Closes #10924 from sebastienrainville/master.
Showing
- core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala 36 additions, 17 deletions...scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala
- core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerUtils.scala 4 additions, 0 deletions...e/spark/scheduler/cluster/mesos/MesosSchedulerUtils.scala
- core/src/test/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackendSuite.scala 13 additions, 0 deletions...uler/cluster/mesos/CoarseMesosSchedulerBackendSuite.scala
Loading
Please register or sign in to comment