Skip to content
Snippets Groups Projects
Commit b6c797b0 authored by Ashwin Shankar's avatar Ashwin Shankar Committed by Sandy Ryza
Browse files

[SPARK-7451] [YARN] Preemption of executors is counted as failure causing Spark job to fail

Added a check to handle container exit status for the preemption scenario, log an INFO message in such cases and move on.
andrewor14

Author: Ashwin Shankar <ashankar@netflix.com>

Closes #5993 from ashwinshankar77/SPARK-7451 and squashes the following commits:

90900cf [Ashwin Shankar] Fix log info message
cf8b6cf [Ashwin Shankar] Stop counting preemption of executors as failure
parent 84bf931f
No related branches found
No related tags found
No related merge requests found
......@@ -373,7 +373,9 @@ private[yarn] class YarnAllocator(
// Hadoop 2.2.X added a ContainerExitStatus we should switch to use
// there are some exit status' we shouldn't necessarily count against us, but for
// now I think its ok as none of the containers are expected to exit
if (completedContainer.getExitStatus == -103) { // vmem limit exceeded
if (completedContainer.getExitStatus == ContainerExitStatus.PREEMPTED) {
logInfo("Container preempted: " + containerId)
} else if (completedContainer.getExitStatus == -103) { // vmem limit exceeded
logWarning(memLimitExceededLogMessage(
completedContainer.getDiagnostics,
VMEM_EXCEEDED_PATTERN))
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment