Skip to content
Snippets Groups Projects
Commit 480357cc authored by petermaxlee's avatar petermaxlee Committed by Reynold Xin
Browse files

[SPARK-16304] LinkageError should not crash Spark executor

## What changes were proposed in this pull request?
This patch updates the failure handling logic so Spark executor does not crash when seeing LinkageError.

## How was this patch tested?
Added an end-to-end test in FailureSuite.

Author: petermaxlee <petermaxlee@gmail.com>

Closes #13982 from petermaxlee/SPARK-16304.
parent 4e14199f
No related branches found
No related tags found
No related merge requests found
......@@ -1881,7 +1881,11 @@ private[spark] object Utils extends Logging {
/** Returns true if the given exception was fatal. See docs for scala.util.control.NonFatal. */
def isFatalError(e: Throwable): Boolean = {
e match {
case NonFatal(_) | _: InterruptedException | _: NotImplementedError | _: ControlThrowable =>
case NonFatal(_) |
_: InterruptedException |
_: NotImplementedError |
_: ControlThrowable |
_: LinkageError =>
false
case _ =>
true
......
......@@ -253,6 +253,15 @@ class FailureSuite extends SparkFunSuite with LocalSparkContext {
rdd.count()
}
test("SPARK-16304: Link error should not crash executor") {
sc = new SparkContext("local[1,2]", "test")
intercept[SparkException] {
sc.parallelize(1 to 2).foreach { i =>
throw new LinkageError()
}
}
}
// TODO: Need to add tests with shuffle fetch failures.
}
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment