Skip to content
Snippets Groups Projects
Commit b38dec2f authored by mohit.goyal's avatar mohit.goyal Committed by Andrew Or
Browse files

[SPARK-5951][YARN] Remove unreachable driver memory properties in yarn client mode

Remove unreachable driver memory properties in yarn client mode

Author: mohit.goyal <mohit.goyal@guavus.com>

Closes #4730 from zuxqoj/master and squashes the following commits:

977dc96 [mohit.goyal] remove not rechable deprecated variables in yarn client mode
parent c871e2da
No related branches found
No related tags found
No related merge requests found
...@@ -79,18 +79,12 @@ private[spark] class YarnClientSchedulerBackend( ...@@ -79,18 +79,12 @@ private[spark] class YarnClientSchedulerBackend(
) )
// Warn against the following deprecated environment variables: env var -> suggestion // Warn against the following deprecated environment variables: env var -> suggestion
val deprecatedEnvVars = Map( val deprecatedEnvVars = Map(
"SPARK_MASTER_MEMORY" -> "SPARK_DRIVER_MEMORY or --driver-memory through spark-submit",
"SPARK_WORKER_INSTANCES" -> "SPARK_WORKER_INSTANCES or --num-executors through spark-submit", "SPARK_WORKER_INSTANCES" -> "SPARK_WORKER_INSTANCES or --num-executors through spark-submit",
"SPARK_WORKER_MEMORY" -> "SPARK_EXECUTOR_MEMORY or --executor-memory through spark-submit", "SPARK_WORKER_MEMORY" -> "SPARK_EXECUTOR_MEMORY or --executor-memory through spark-submit",
"SPARK_WORKER_CORES" -> "SPARK_EXECUTOR_CORES or --executor-cores through spark-submit") "SPARK_WORKER_CORES" -> "SPARK_EXECUTOR_CORES or --executor-cores through spark-submit")
// Do the same for deprecated properties: property -> suggestion
val deprecatedProps = Map("spark.master.memory" -> "--driver-memory through spark-submit")
optionTuples.foreach { case (optionName, envVar, sparkProp) => optionTuples.foreach { case (optionName, envVar, sparkProp) =>
if (sc.getConf.contains(sparkProp)) { if (sc.getConf.contains(sparkProp)) {
extraArgs += (optionName, sc.getConf.get(sparkProp)) extraArgs += (optionName, sc.getConf.get(sparkProp))
if (deprecatedProps.contains(sparkProp)) {
logWarning(s"NOTE: $sparkProp is deprecated. Use ${deprecatedProps(sparkProp)} instead.")
}
} else if (System.getenv(envVar) != null) { } else if (System.getenv(envVar) != null) {
extraArgs += (optionName, System.getenv(envVar)) extraArgs += (optionName, System.getenv(envVar))
if (deprecatedEnvVars.contains(envVar)) { if (deprecatedEnvVars.contains(envVar)) {
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment