Skip to content
Snippets Groups Projects
Commit 6dd64e8b authored by Y.CORP.YAHOO.COM\tgraves's avatar Y.CORP.YAHOO.COM\tgraves
Browse files

Update docs and remove old reference to --user option

parent dfb4c697
No related branches found
No related tags found
No related merge requests found
...@@ -57,7 +57,6 @@ The command to launch the YARN Client is as follows: ...@@ -57,7 +57,6 @@ The command to launch the YARN Client is as follows:
--master-memory <MEMORY_FOR_MASTER> \ --master-memory <MEMORY_FOR_MASTER> \
--worker-memory <MEMORY_PER_WORKER> \ --worker-memory <MEMORY_PER_WORKER> \
--worker-cores <CORES_PER_WORKER> \ --worker-cores <CORES_PER_WORKER> \
--user <hadoop_user> \
--queue <queue_name> --queue <queue_name>
For example: For example:
...@@ -77,5 +76,4 @@ The above starts a YARN Client programs which periodically polls the Application ...@@ -77,5 +76,4 @@ The above starts a YARN Client programs which periodically polls the Application
- When your application instantiates a Spark context it must use a special "yarn-standalone" master url. This starts the scheduler without forcing it to connect to a cluster. A good way to handle this is to pass "yarn-standalone" as an argument to your program, as shown in the example above. - When your application instantiates a Spark context it must use a special "yarn-standalone" master url. This starts the scheduler without forcing it to connect to a cluster. A good way to handle this is to pass "yarn-standalone" as an argument to your program, as shown in the example above.
- We do not requesting container resources based on the number of cores. Thus the numbers of cores given via command line arguments cannot be guaranteed. - We do not requesting container resources based on the number of cores. Thus the numbers of cores given via command line arguments cannot be guaranteed.
- Currently, we have not yet integrated with hadoop security. If --user is present, the hadoop_user specified will be used to run the tasks on the cluster. If unspecified, current user will be used (which should be valid in cluster). - The local directories used for spark will be the local directories configured for YARN (Hadoop Yarn config yarn.nodemanager.local-dirs). If the user specifies spark.local.dir, it will be ignored.
Once hadoop security support is added, and if hadoop cluster is enabled with security, additional restrictions would apply via delegation tokens passed.
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment