Skip to content
Snippets Groups Projects
user avatar
Stavros Kontopoulos authored
## What changes were proposed in this pull request?
Adds a sandbox link per driver in the dispatcher ui with minimal changes after a bug was fixed here:
https://issues.apache.org/jira/browse/MESOS-4992
The sandbox uri has the following format:
http://<proxy_uri>/#/slaves/\<agent-id\>/ frameworks/ \<scheduler-id\>/executors/\<driver-id\>/browse

For dc/os the proxy uri is <dc/os uri>/mesos. For the dc/os deployment scenario and to make things easier I introduced a new config property named `spark.mesos.proxy.baseURL` which should be passed to the dispatcher when launched using --conf. If no such configuration is detected then no sandbox uri is depicted, and there is an empty column with a header (this can be changed so nothing is shown).

Within dc/os the base url must be a property for the dispatcher that we should add in the future here:
https://github.com/mesosphere/universe/blob/9e7c909c3b8680eeb0494f2a58d5746e3bab18c1/repo/packages/S/spark/26/config.json
It is not easy to detect in different environments what is that uri so user should pass it.

## How was this patch tested?
Tested with the mesos test suite here: https://github.com/typesafehub/mesos-spark-integration-tests.
Attached image shows the ui modification where the sandbox header is added.
![image](https://user-images.githubusercontent.com/7945591/27831630-2a3b447e-60d4-11e7-87bb-d057efd4efa7.png)

Tested the uri redirection the way it was suggested here:
https://issues.apache.org/jira/browse/MESOS-4992

Built mesos 1.4 from the master branch and started the mesos dispatcher with the command:

`./sbin/start-mesos-dispatcher.sh --conf spark.mesos.proxy.baseURL=http://localhost:5050 -m mesos://127.0.0.1:5050`

Run a spark example:

`./bin/spark-submit   --class org.apache.spark.examples.SparkPi   --master mesos://10.10.1.79:7078   --deploy-mode cluster   --executor-memory 2G   --total-executor-cores 2     http://<path>/spark-examples_2.11-2.1.1.jar  10`

Sandbox uri is shown at the bottom of the page:

![image](https://user-images.githubusercontent.com/7945591/28599237-89d0a8c8-71b1-11e7-8f94-41ad117ceead.png)

Redirection works as expected:
![image](https://user-images.githubusercontent.com/7945591/28599247-a5d65248-71b1-11e7-8b5e-a0ac2a79fa23.png)

Author: Stavros Kontopoulos <st.kontopoulos@gmail.com>

Closes #18528 from skonto/adds_the_sandbox_uri.
663f30d1
History
Name Last commit Last update
..
_data
_includes
_layouts
_plugins
css
img
js
README.md
_config.yml
api.md
building-spark.md
cloud-integration.md
cluster-overview.md
configuration.md
contributing-to-spark.md
graphx-programming-guide.md
hadoop-provided.md
hardware-provisioning.md
index.md
job-scheduling.md
ml-advanced.md
ml-ann.md
ml-classification-regression.md
ml-clustering.md
ml-collaborative-filtering.md
ml-decision-tree.md
ml-ensembles.md
ml-features.md
ml-frequent-pattern-mining.md
ml-guide.md
ml-linear-methods.md
ml-migration-guides.md
ml-pipeline.md
ml-statistics.md
ml-survival-regression.md
ml-tuning.md
mllib-classification-regression.md
mllib-clustering.md
mllib-collaborative-filtering.md
mllib-data-types.md
mllib-decision-tree.md
mllib-dimensionality-reduction.md
mllib-ensembles.md
mllib-evaluation-metrics.md
mllib-feature-extraction.md
mllib-frequent-pattern-mining.md
mllib-guide.md
mllib-isotonic-regression.md
mllib-linear-methods.md
mllib-migration-guides.md
mllib-naive-bayes.md
mllib-optimization.md
mllib-pmml-model-export.md
mllib-statistics.md
monitoring.md
programming-guide.md
quick-start.md
rdd-programming-guide.md
running-on-mesos.md
running-on-yarn.md
security.md
spark-standalone.md
sparkr.md
sql-programming-guide.md
storage-openstack-swift.md
streaming-custom-receivers.md
streaming-flume-integration.md
streaming-kafka-0-10-integration.md
streaming-kafka-0-8-integration.md
streaming-kafka-integration.md
streaming-kinesis-integration.md
streaming-programming-guide.md
structured-streaming-kafka-integration.md
structured-streaming-programming-guide.md
submitting-applications.md
tuning.md