Skip to content
Snippets Groups Projects
Commit 6c20f352 authored by Andrew Or's avatar Andrew Or
Browse files

[SPARK-6133] Make sc.stop() idempotent

Before we would get the following (benign) error if we called `sc.stop()` twice. This is because the listener bus would try to post the end event again even after it has already stopped. This happens occasionally when flaky tests fail, usually as a result of other sources of error. Either way we shouldn't be logging this error when it is not the cause of the failure.
```
ERROR LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerApplicationEnd(1425348445682)
```

Author: Andrew Or <andrew@databricks.com>

Closes #4871 from andrewor14/sc-stop and squashes the following commits:

a14afc5 [Andrew Or] Move code after code
915db16 [Andrew Or] Move code into code
parent fe63e822
No related branches found
No related tags found
No related merge requests found
...@@ -1392,10 +1392,10 @@ class SparkContext(config: SparkConf) extends Logging with ExecutorAllocationCli ...@@ -1392,10 +1392,10 @@ class SparkContext(config: SparkConf) extends Logging with ExecutorAllocationCli
/** Shut down the SparkContext. */ /** Shut down the SparkContext. */
def stop() { def stop() {
SparkContext.SPARK_CONTEXT_CONSTRUCTOR_LOCK.synchronized { SparkContext.SPARK_CONTEXT_CONSTRUCTOR_LOCK.synchronized {
postApplicationEnd()
ui.foreach(_.stop())
if (!stopped) { if (!stopped) {
stopped = true stopped = true
postApplicationEnd()
ui.foreach(_.stop())
env.metricsSystem.report() env.metricsSystem.report()
metadataCleaner.cancel() metadataCleaner.cancel()
cleaner.foreach(_.stop()) cleaner.foreach(_.stop())
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment