Skip to content
Snippets Groups Projects
Commit 39759ff0 authored by Nicholas Chammas's avatar Nicholas Chammas Committed by Marcelo Vanzin
Browse files

[DOCS][MINOR] Update location of Spark YARN shuffle jar


Looking at the distributions provided on spark.apache.org, I see that the Spark YARN shuffle jar is under `yarn/` and not `lib/`.

This change is so minor I'm not sure it needs a JIRA. But let me know if so and I'll create one.

Author: Nicholas Chammas <nicholas.chammas@gmail.com>

Closes #16130 from nchammas/yarn-doc-fix.

(cherry picked from commit 5a92dc76)
Signed-off-by: default avatarMarcelo Vanzin <vanzin@cloudera.com>
parent e23c8cfc
No related branches found
No related tags found
No related merge requests found
...@@ -518,7 +518,7 @@ instructions: ...@@ -518,7 +518,7 @@ instructions:
pre-packaged distribution. pre-packaged distribution.
1. Locate the `spark-<version>-yarn-shuffle.jar`. This should be under 1. Locate the `spark-<version>-yarn-shuffle.jar`. This should be under
`$SPARK_HOME/common/network-yarn/target/scala-<version>` if you are building Spark yourself, and under `$SPARK_HOME/common/network-yarn/target/scala-<version>` if you are building Spark yourself, and under
`lib` if you are using a distribution. `yarn` if you are using a distribution.
1. Add this jar to the classpath of all `NodeManager`s in your cluster. 1. Add this jar to the classpath of all `NodeManager`s in your cluster.
1. In the `yarn-site.xml` on each node, add `spark_shuffle` to `yarn.nodemanager.aux-services`, 1. In the `yarn-site.xml` on each node, add `spark_shuffle` to `yarn.nodemanager.aux-services`,
then set `yarn.nodemanager.aux-services.spark_shuffle.class` to then set `yarn.nodemanager.aux-services.spark_shuffle.class` to
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment