Skip to content
Snippets Groups Projects
Commit 05c9aa9e authored by Patrick Wendell's avatar Patrick Wendell Committed by Aaron Davidson
Browse files

SPARK-1652: Set driver memory correctly in spark-submit.

The previous check didn't account for the fact that the default
deploy mode is "client" unless otherwise specified. Also, this
sets the more narrowly defined SPARK_DRIVER_MEMORY instead of setting
SPARK_MEM.

Author: Patrick Wendell <pwendell@gmail.com>

Closes #730 from pwendell/spark-submit and squashes the following commits:

430b98f [Patrick Wendell] Feedback from Aaron
e788edf [Patrick Wendell] Changes based on Aaron's feedback
f508146 [Patrick Wendell] SPARK-1652: Set driver memory correctly in spark-submit.
parent 7d9cc921
No related branches found
No related tags found
No related merge requests found
...@@ -35,8 +35,10 @@ while (($#)); do ...@@ -35,8 +35,10 @@ while (($#)); do
shift shift
done done
if [ ! -z $DRIVER_MEMORY ] && [ ! -z $DEPLOY_MODE ] && [ $DEPLOY_MODE = "client" ]; then DEPLOY_MODE=${DEPLOY_MODE:-"client"}
export SPARK_MEM=$DRIVER_MEMORY
if [ -n "$DRIVER_MEMORY" ] && [ $DEPLOY_MODE == "client" ]; then
export SPARK_DRIVER_MEMORY=$DRIVER_MEMORY
fi fi
$SPARK_HOME/bin/spark-class org.apache.spark.deploy.SparkSubmit "${ORIG_ARGS[@]}" $SPARK_HOME/bin/spark-class org.apache.spark.deploy.SparkSubmit "${ORIG_ARGS[@]}"
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment