Skip to content
Snippets Groups Projects
Commit d850b4bd authored by jerryshao's avatar jerryshao Committed by Andrew Or
Browse files

[SPARK-6975][Yarn] Fix argument validation error

`numExecutors` checking is failed when dynamic allocation is enabled with default configuration. Details can be seen is [SPARK-6975](https://issues.apache.org/jira/browse/SPARK-6975). sryza, please help me to review this, not sure is this the correct way, I think previous you change this part :)

Author: jerryshao <saisai.shao@intel.com>

Closes #5551 from jerryshao/SPARK-6975 and squashes the following commits:

4335da1 [jerryshao] Change according to the comments
77bdcbd [jerryshao] Fix argument validation error
parent 19913373
No related branches found
No related tags found
No related merge requests found
......@@ -103,9 +103,13 @@ private[spark] class ClientArguments(args: Array[String], sparkConf: SparkConf)
* This is intended to be called only after the provided arguments have been parsed.
*/
private def validateArgs(): Unit = {
if (numExecutors <= 0) {
if (numExecutors < 0 || (!isDynamicAllocationEnabled && numExecutors == 0)) {
throw new IllegalArgumentException(
"You must specify at least 1 executor!\n" + getUsageMessage())
s"""
|Number of executors was $numExecutors, but must be at least 1
|(or 0 if dynamic executor allocation is enabled).
|${getUsageMessage()}
""".stripMargin)
}
if (executorCores < sparkConf.getInt("spark.task.cpus", 1)) {
throw new SparkException("Executor cores must not be less than " +
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment