Skip to content
Snippets Groups Projects
Commit 06dc4b52 authored by Zhang, Liye's avatar Zhang, Liye Committed by Andrew Or
Browse files

[SPARK-4989][CORE] avoid wrong eventlog conf cause cluster down in standalone mode

when enabling eventlog in standalone mode, if give the wrong configuration, the standalone cluster will down (cause master restart, lose connection with workers).
How to reproduce: just give an invalid value to "spark.eventLog.dir", for example: spark.eventLog.dir=hdfs://tmp/logdir1, hdfs://tmp/logdir2. This will throw illegalArgumentException, which will cause the Master restart. And the whole cluster is not available.

Author: Zhang, Liye <liye.zhang@intel.com>

Closes #3824 from liyezhang556520/wrongConf4Cluster and squashes the following commits:

3c24d98 [Zhang, Liye] revert change with logwarning and excetption for FileNotFoundException
3c1ac2e [Zhang, Liye] change var to val
a49c52f [Zhang, Liye] revert wrong modification
12eee85 [Zhang, Liye] add more message in log and on webUI
5c1fa33 [Zhang, Liye] cache exceptions when eventlog with wrong conf
parent f825e193
No related branches found
No related tags found
No related merge requests found
...@@ -720,26 +720,27 @@ private[spark] class Master( ...@@ -720,26 +720,27 @@ private[spark] class Master(
def rebuildSparkUI(app: ApplicationInfo): Boolean = { def rebuildSparkUI(app: ApplicationInfo): Boolean = {
val appName = app.desc.name val appName = app.desc.name
val notFoundBasePath = HistoryServer.UI_PATH_PREFIX + "/not-found" val notFoundBasePath = HistoryServer.UI_PATH_PREFIX + "/not-found"
val eventLogFile = app.desc.eventLogDir try {
.map { dir => EventLoggingListener.getLogPath(dir, app.id) } val eventLogFile = app.desc.eventLogDir
.getOrElse { .map { dir => EventLoggingListener.getLogPath(dir, app.id) }
// Event logging is not enabled for this application .getOrElse {
app.desc.appUiUrl = notFoundBasePath // Event logging is not enabled for this application
return false app.desc.appUiUrl = notFoundBasePath
} return false
val fs = Utils.getHadoopFileSystem(eventLogFile, hadoopConf) }
val fs = Utils.getHadoopFileSystem(eventLogFile, hadoopConf)
if (fs.exists(new Path(eventLogFile + EventLoggingListener.IN_PROGRESS))) { if (fs.exists(new Path(eventLogFile + EventLoggingListener.IN_PROGRESS))) {
// Event logging is enabled for this application, but the application is still in progress // Event logging is enabled for this application, but the application is still in progress
val title = s"Application history not found (${app.id})" val title = s"Application history not found (${app.id})"
var msg = s"Application $appName is still in progress." var msg = s"Application $appName is still in progress."
logWarning(msg) logWarning(msg)
msg = URLEncoder.encode(msg, "UTF-8") msg = URLEncoder.encode(msg, "UTF-8")
app.desc.appUiUrl = notFoundBasePath + s"?msg=$msg&title=$title" app.desc.appUiUrl = notFoundBasePath + s"?msg=$msg&title=$title"
return false return false
} }
try {
val (logInput, sparkVersion) = EventLoggingListener.openEventLog(new Path(eventLogFile), fs) val (logInput, sparkVersion) = EventLoggingListener.openEventLog(new Path(eventLogFile), fs)
val replayBus = new ReplayListenerBus() val replayBus = new ReplayListenerBus()
val ui = SparkUI.createHistoryUI(new SparkConf, replayBus, new SecurityManager(conf), val ui = SparkUI.createHistoryUI(new SparkConf, replayBus, new SecurityManager(conf),
...@@ -758,7 +759,7 @@ private[spark] class Master( ...@@ -758,7 +759,7 @@ private[spark] class Master(
case fnf: FileNotFoundException => case fnf: FileNotFoundException =>
// Event logging is enabled for this application, but no event logs are found // Event logging is enabled for this application, but no event logs are found
val title = s"Application history not found (${app.id})" val title = s"Application history not found (${app.id})"
var msg = s"No event logs found for application $appName in $eventLogFile." var msg = s"No event logs found for application $appName in ${app.desc.eventLogDir}."
logWarning(msg) logWarning(msg)
msg += " Did you specify the correct logging directory?" msg += " Did you specify the correct logging directory?"
msg = URLEncoder.encode(msg, "UTF-8") msg = URLEncoder.encode(msg, "UTF-8")
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment