-
- Downloads
[SPARK-4138][SPARK-4139] Improve dynamic allocation settings
This should be merged after #2746 (SPARK-3795). **SPARK-4138**. If the user sets both the number of executors and `spark.dynamicAllocation.enabled`, we should throw an exception. **SPARK-4139**. If the user sets `spark.dynamicAllocation.enabled`, we should use the max number of executors as the starting number of executors because the first job is likely to run immediately after application startup. If the latter is not set, throw an exception. Author: Andrew Or <andrew@databricks.com> Closes #3002 from andrewor14/yarn-set-executors and squashes the following commits: c528fce [Andrew Or] Merge branch 'master' of github.com:apache/spark into yarn-set-executors 55d4699 [Andrew Or] Bug fix: `isDynamicAllocationEnabled` was always false 2b0ccec [Andrew Or] Start the number of executors at the max 022bfde [Andrew Or] Guard against incompatible settings of number of executors
Showing
- yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMasterArguments.scala 2 additions, 1 deletion...apache/spark/deploy/yarn/ApplicationMasterArguments.scala
- yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala 23 additions, 7 deletions.../scala/org/apache/spark/deploy/yarn/ClientArguments.scala
- yarn/common/src/main/scala/org/apache/spark/deploy/yarn/YarnSparkHadoopUtil.scala 2 additions, 0 deletions...la/org/apache/spark/deploy/yarn/YarnSparkHadoopUtil.scala
- yarn/common/src/main/scala/org/apache/spark/scheduler/cluster/YarnClusterSchedulerBackend.scala 2 additions, 2 deletions...spark/scheduler/cluster/YarnClusterSchedulerBackend.scala
Loading
Please register or sign in to comment