Skip to content
Snippets Groups Projects
Commit b1bd1dd3 authored by Davies Liu's avatar Davies Liu Committed by Josh Rosen
Browse files

[SPARK-5788] [PySpark] capture the exception in python write thread

The exception in Python writer thread will shutdown executor.

Author: Davies Liu <davies@databricks.com>

Closes #4577 from davies/exception and squashes the following commits:

eb0ceff [Davies Liu] Update PythonRDD.scala
139b0db [Davies Liu] capture the exception in python write thread
parent 1294a6e0
No related branches found
No related tags found
No related merge requests found
......@@ -248,13 +248,13 @@ private[spark] class PythonRDD(
} catch {
case e: Exception if context.isCompleted || context.isInterrupted =>
logDebug("Exception thrown after task completion (likely due to cleanup)", e)
worker.shutdownOutput()
Utils.tryLog(worker.shutdownOutput())
case e: Exception =>
// We must avoid throwing exceptions here, because the thread uncaught exception handler
// will kill the whole executor (see org.apache.spark.executor.Executor).
_exception = e
worker.shutdownOutput()
Utils.tryLog(worker.shutdownOutput())
} finally {
// Release memory used by this thread for shuffles
env.shuffleMemoryManager.releaseMemoryForThisThread()
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment