-
- Downloads
[SPARK-7736] [CORE] [YARN] Make pyspark fail YARN app on failure.
The YARN backend doesn't like when user code calls `System.exit`, since it cannot know the exit status and thus cannot set an appropriate final status for the application. So, for pyspark, avoid that call and instead throw an exception with the exit code. SparkSubmit handles that exception and exits with the given exit code, while YARN uses the exit code as the failure code for the Spark app. Author: Marcelo Vanzin <vanzin@cloudera.com> Closes #7751 from vanzin/SPARK-9416.
Showing
- core/src/main/scala/org/apache/spark/SparkException.scala 7 additions, 0 deletionscore/src/main/scala/org/apache/spark/SparkException.scala
- core/src/main/scala/org/apache/spark/deploy/PythonRunner.scala 19 additions, 4 deletions...src/main/scala/org/apache/spark/deploy/PythonRunner.scala
- core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala 8 additions, 2 deletions.../src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
- yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala 6 additions, 2 deletions...cala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
Loading
Please register or sign in to comment