-
- Downloads
[SPARK-16637] Unified containerizer
## What changes were proposed in this pull request? New config var: spark.mesos.docker.containerizer={"mesos","docker" (default)} This adds support for running docker containers via the Mesos unified containerizer: http://mesos.apache.org/documentation/latest/container-image/ The benefit is losing the dependency on `dockerd`, and all the costs which it incurs. I've also updated the supported Mesos version to 0.28.2 for support of the required protobufs. This is blocked on: https://github.com/apache/spark/pull/14167 ## How was this patch tested? - manually testing jobs submitted with both "mesos" and "docker" settings for the new config var. - spark/mesos integration test suite Author: Michael Gummelt <mgummelt@mesosphere.io> Closes #14275 from mgummelt/unified-containerizer.
Showing
- core/src/main/scala/org/apache/spark/SparkConf.scala 9 additions, 4 deletionscore/src/main/scala/org/apache/spark/SparkConf.scala
- core/src/main/scala/org/apache/spark/TaskState.scala 3 additions, 5 deletionscore/src/main/scala/org/apache/spark/TaskState.scala
- core/src/main/scala/org/apache/spark/deploy/mesos/MesosDriverDescription.scala 8 additions, 3 deletions...rg/apache/spark/deploy/mesos/MesosDriverDescription.scala
- core/src/main/scala/org/apache/spark/deploy/mesos/ui/DriverPage.scala 1 addition, 1 deletion...n/scala/org/apache/spark/deploy/mesos/ui/DriverPage.scala
- core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala 45 additions, 40 deletions...spark/scheduler/cluster/mesos/MesosClusterScheduler.scala
- core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackend.scala 1 addition, 1 deletion...er/cluster/mesos/MesosCoarseGrainedSchedulerBackend.scala
- core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosFineGrainedSchedulerBackend.scala 1 addition, 1 deletion...uler/cluster/mesos/MesosFineGrainedSchedulerBackend.scala
- core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackendUtil.scala 33 additions, 13 deletions...k/scheduler/cluster/mesos/MesosSchedulerBackendUtil.scala
- core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerUtils.scala 2 additions, 1 deletion...e/spark/scheduler/cluster/mesos/MesosSchedulerUtils.scala
- core/src/test/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackendSuite.scala 29 additions, 3 deletions...uster/mesos/MesosCoarseGrainedSchedulerBackendSuite.scala
- dev/deps/spark-deps-hadoop-2.2 1 addition, 1 deletiondev/deps/spark-deps-hadoop-2.2
- dev/deps/spark-deps-hadoop-2.3 1 addition, 1 deletiondev/deps/spark-deps-hadoop-2.3
- dev/deps/spark-deps-hadoop-2.4 1 addition, 1 deletiondev/deps/spark-deps-hadoop-2.4
- dev/deps/spark-deps-hadoop-2.6 1 addition, 1 deletiondev/deps/spark-deps-hadoop-2.6
- dev/deps/spark-deps-hadoop-2.7 1 addition, 1 deletiondev/deps/spark-deps-hadoop-2.7
- docs/_config.yml 1 addition, 1 deletiondocs/_config.yml
- docs/running-on-mesos.md 10 additions, 0 deletionsdocs/running-on-mesos.md
- pom.xml 1 addition, 1 deletionpom.xml
Loading
Please register or sign in to comment