Skip to content
Snippets Groups Projects
Commit 801e4d09 authored by Sean Owen's avatar Sean Owen
Browse files

[SPARK-16606][CORE] Misleading warning for SparkContext.getOrCreate "WARN...

[SPARK-16606][CORE] Misleading warning for SparkContext.getOrCreate "WARN SparkContext: Use an existing SparkContext, some configuration may not take effect."

## What changes were proposed in this pull request?

SparkContext.getOrCreate shouldn't warn about ignored config if

- it wasn't ignored because a new context is created with it or
- no config was actually provided

## How was this patch tested?

Jenkins + existing tests.

Author: Sean Owen <sowen@cloudera.com>

Closes #14533 from srowen/SPARK-16606.
parent bb2b9d0a
No related branches found
No related tags found
No related merge requests found
......@@ -2262,9 +2262,10 @@ object SparkContext extends Logging {
SPARK_CONTEXT_CONSTRUCTOR_LOCK.synchronized {
if (activeContext.get() == null) {
setActiveContext(new SparkContext(config), allowMultipleContexts = false)
}
if (config.getAll.nonEmpty) {
logWarning("Use an existing SparkContext, some configuration may not take effect.")
} else {
if (config.getAll.nonEmpty) {
logWarning("Using an existing SparkContext; some configuration may not take effect.")
}
}
activeContext.get()
}
......@@ -2281,7 +2282,12 @@ object SparkContext extends Logging {
* even if multiple contexts are allowed.
*/
def getOrCreate(): SparkContext = {
getOrCreate(new SparkConf())
SPARK_CONTEXT_CONSTRUCTOR_LOCK.synchronized {
if (activeContext.get() == null) {
setActiveContext(new SparkContext(), allowMultipleContexts = false)
}
activeContext.get()
}
}
/**
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment