Skip to content
Snippets Groups Projects
Commit 86fa4ba6 authored by Yadong Qi's avatar Yadong Qi Committed by Sean Owen
Browse files

[SPARK-9737] [YARN] Add the suggested configuration when required executor...

[SPARK-9737] [YARN] Add the suggested configuration when required executor memory is above the max threshold of this cluster on YARN mode

Author: Yadong Qi <qiyadong2010@gmail.com>

Closes #8028 from watermen/SPARK-9737 and squashes the following commits:

48bdf3d [Yadong Qi] Add suggested configuration.
parent 68ccc6e1
No related branches found
No related tags found
No related merge requests found
......@@ -203,12 +203,14 @@ private[spark] class Client(
val executorMem = args.executorMemory + executorMemoryOverhead
if (executorMem > maxMem) {
throw new IllegalArgumentException(s"Required executor memory (${args.executorMemory}" +
s"+$executorMemoryOverhead MB) is above the max threshold ($maxMem MB) of this cluster!")
s"+$executorMemoryOverhead MB) is above the max threshold ($maxMem MB) of this cluster! " +
"Please increase the value of 'yarn.scheduler.maximum-allocation-mb'.")
}
val amMem = args.amMemory + amMemoryOverhead
if (amMem > maxMem) {
throw new IllegalArgumentException(s"Required AM memory (${args.amMemory}" +
s"+$amMemoryOverhead MB) is above the max threshold ($maxMem MB) of this cluster!")
s"+$amMemoryOverhead MB) is above the max threshold ($maxMem MB) of this cluster! " +
"Please increase the value of 'yarn.scheduler.maximum-allocation-mb'.")
}
logInfo("Will allocate AM container, with %d MB memory including %d MB overhead".format(
amMem,
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment