-
- Downloads
[SPARK-4012] stop SparkContext when the exception is thrown from an infinite loop
https://issues.apache.org/jira/browse/SPARK-4012 This patch is a resubmission for https://github.com/apache/spark/pull/2864 What I am proposing in this patch is that ***when the exception is thrown from an infinite loop, we should stop the SparkContext, instead of let JVM throws exception forever*** So, in the infinite loops where we originally wrapped with a ` logUncaughtExceptions`, I changed to `tryOrStopSparkContext`, so that the Spark component is stopped Early stopped JVM process is helpful for HA scheme design, for example, The user has a script checking the existence of the pid of the Spark Streaming driver for monitoring the availability; with the code before this patch, the JVM process is still available but not functional when the exceptions are thrown andrewor14, srowen , mind taking further consideration about the change? Author: CodingCat <zhunansjtu@gmail.com> Closes #5004 from CodingCat/SPARK-4012-1 and squashes the following commits: 589276a [CodingCat] throw fatal error again 3c72cd8 [CodingCat] address the comments 6087864 [CodingCat] revise comments 6ad3eb0 [CodingCat] stop SparkContext instead of quit the JVM process 6322959 [CodingCat] exit JVM process when the exception is thrown from an infinite loop
Showing
- core/src/main/scala/org/apache/spark/ContextCleaner.scala 1 addition, 1 deletioncore/src/main/scala/org/apache/spark/ContextCleaner.scala
- core/src/main/scala/org/apache/spark/SparkContext.scala 1 addition, 1 deletioncore/src/main/scala/org/apache/spark/SparkContext.scala
- core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala 1 addition, 1 deletion...a/org/apache/spark/deploy/history/FsHistoryProvider.scala
- core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala 1 addition, 1 deletion.../scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
- core/src/main/scala/org/apache/spark/util/AsynchronousListenerBus.scala 8 additions, 2 deletions...scala/org/apache/spark/util/AsynchronousListenerBus.scala
- core/src/main/scala/org/apache/spark/util/Utils.scala 28 additions, 0 deletionscore/src/main/scala/org/apache/spark/util/Utils.scala
- core/src/test/scala/org/apache/spark/scheduler/EventLoggingListenerSuite.scala 5 additions, 4 deletions...rg/apache/spark/scheduler/EventLoggingListenerSuite.scala
- core/src/test/scala/org/apache/spark/scheduler/SparkListenerSuite.scala 5 additions, 5 deletions...scala/org/apache/spark/scheduler/SparkListenerSuite.scala
- streaming/src/main/scala/org/apache/spark/streaming/scheduler/JobScheduler.scala 1 addition, 1 deletion...a/org/apache/spark/streaming/scheduler/JobScheduler.scala
Loading
Please register or sign in to comment