Skip to content
Snippets Groups Projects
Commit 041eccb4 authored by Sital Kedia's avatar Sital Kedia Committed by Marcelo Vanzin
Browse files

[SPARK-21834] Incorrect executor request in case of dynamic allocation

## What changes were proposed in this pull request?

killExecutor api currently does not allow killing an executor without updating the total number of executors needed. In case of dynamic allocation is turned on and the allocator tries to kill an executor, the scheduler reduces the total number of executors needed ( see https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala#L635

) which is incorrect because the allocator already takes care of setting the required number of executors itself.

## How was this patch tested?

Ran a job on the cluster and made sure the executor request is correct

Author: Sital Kedia <skedia@fb.com>

Closes #19081 from sitalkedia/skedia/oss_fix_executor_allocation.

(cherry picked from commit 6949a9c5)
Signed-off-by: default avatarMarcelo Vanzin <vanzin@cloudera.com>
parent 57697535
No related branches found
No related tags found
No related merge requests found
......@@ -427,6 +427,9 @@ private[spark] class ExecutorAllocationManager(
} else {
client.killExecutors(executorIdsToBeRemoved)
}
// [SPARK-21834] killExecutors api reduces the target number of executors.
// So we need to update the target with desired value.
client.requestTotalExecutors(numExecutorsTarget, localityAwareTasks, hostToLocalTaskCount)
// reset the newExecutorTotal to the existing number of executors
newExecutorTotal = numExistingExecutors
if (testing || executorsRemoved.nonEmpty) {
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment