-
- Downloads
[SPARK-4286] Add an external shuffle service that can be run as a daemon.
This allows Mesos deployments to use the shuffle service (and implicitly dynamic allocation). It does so by adding a new "main" class and two corresponding scripts in `sbin`: - `sbin/start-shuffle-service.sh` - `sbin/stop-shuffle-service.sh` Specific options can be passed in `SPARK_SHUFFLE_OPTS`. This is picking up work from #3861 /cc tnachen Author: Iulian Dragos <jaguarul@gmail.com> Closes #4990 from dragos/feature/external-shuffle-service and squashes the following commits: 6c2b148 [Iulian Dragos] Import order and wrong name fixup. 07804ad [Iulian Dragos] Moved ExternalShuffleService to the `deploy` package + other minor tweaks. 4dc1f91 [Iulian Dragos] Reviewer’s comments: 8145429 [Iulian Dragos] Add an external shuffle service that can be run as a daemon.
Showing
- conf/spark-env.sh.template 2 additions, 1 deletionconf/spark-env.sh.template
- core/src/main/scala/org/apache/spark/deploy/ExternalShuffleService.scala 52 additions, 7 deletions...cala/org/apache/spark/deploy/ExternalShuffleService.scala
- core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala 7 additions, 6 deletions...rc/main/scala/org/apache/spark/deploy/worker/Worker.scala
- docs/job-scheduling.md 1 addition, 1 deletiondocs/job-scheduling.md
- launcher/src/main/java/org/apache/spark/launcher/SparkClassCommandBuilder.java 4 additions, 0 deletions...a/org/apache/spark/launcher/SparkClassCommandBuilder.java
- sbin/start-shuffle-service.sh 33 additions, 0 deletionssbin/start-shuffle-service.sh
- sbin/stop-shuffle-service.sh 25 additions, 0 deletionssbin/stop-shuffle-service.sh
sbin/start-shuffle-service.sh
0 → 100755
sbin/stop-shuffle-service.sh
0 → 100755
Please register or sign in to comment