-
- Downloads
[SPARK-15271][MESOS] Allow force pulling executor docker images
## What changes were proposed in this pull request? (Please fill in changes proposed in this fix) ## How was this patch tested? (Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests) (If this patch involves UI changes, please attach a screenshot; otherwise, remove this) Mesos agents by default will not pull docker images which are cached locally already. In order to run Spark executors from mutable tags like `:latest` this commit introduces a Spark setting (`spark.mesos.executor.docker.forcePullImage`). Setting this flag to true will tell the Mesos agent to force pull the docker image (default is `false` which is consistent with the previous implementation and Mesos' default behaviour). Author: Philipp Hoffmann <mail@philipphoffmann.de> Closes #14348 from philipphoffmann/force-pull-image.
Showing
- core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala 4 additions, 10 deletions...spark/scheduler/cluster/mesos/MesosClusterScheduler.scala
- core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackend.scala 5 additions, 2 deletions...er/cluster/mesos/MesosCoarseGrainedSchedulerBackend.scala
- core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosFineGrainedSchedulerBackend.scala 5 additions, 2 deletions...uler/cluster/mesos/MesosFineGrainedSchedulerBackend.scala
- core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackendUtil.scala 12 additions, 8 deletions...k/scheduler/cluster/mesos/MesosSchedulerBackendUtil.scala
- core/src/test/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackendSuite.scala 63 additions, 0 deletions...uster/mesos/MesosCoarseGrainedSchedulerBackendSuite.scala
- core/src/test/scala/org/apache/spark/scheduler/cluster/mesos/MesosFineGrainedSchedulerBackendSuite.scala 2 additions, 0 deletions...cluster/mesos/MesosFineGrainedSchedulerBackendSuite.scala
- dev/deps/spark-deps-hadoop-2.2 1 addition, 1 deletiondev/deps/spark-deps-hadoop-2.2
- dev/deps/spark-deps-hadoop-2.3 1 addition, 1 deletiondev/deps/spark-deps-hadoop-2.3
- dev/deps/spark-deps-hadoop-2.4 1 addition, 1 deletiondev/deps/spark-deps-hadoop-2.4
- dev/deps/spark-deps-hadoop-2.6 1 addition, 1 deletiondev/deps/spark-deps-hadoop-2.6
- dev/deps/spark-deps-hadoop-2.7 1 addition, 1 deletiondev/deps/spark-deps-hadoop-2.7
- docs/_config.yml 1 addition, 1 deletiondocs/_config.yml
- docs/running-on-mesos.md 12 additions, 0 deletionsdocs/running-on-mesos.md
- pom.xml 1 addition, 1 deletionpom.xml
Loading
Please register or sign in to comment