Skip to content
Snippets Groups Projects
Commit 80098109 authored by Zhang, Liye's avatar Zhang, Liye Committed by Andrew Or
Browse files

[SPARK-6314] [CORE] handle JsonParseException for history server

This is handled in the same way with [SPARK-6197](https://issues.apache.org/jira/browse/SPARK-6197). The result of this PR is that exception showed in history server log will be replaced by a warning, and the application that with un-complete history log file will be listed on history server webUI

Author: Zhang, Liye <liye.zhang@intel.com>

Closes #5736 from liyezhang556520/SPARK-6314 and squashes the following commits:

b8d2d88 [Zhang, Liye] handle JsonParseException for history server
parent 2d222fb3
No related branches found
No related tags found
No related merge requests found
...@@ -333,8 +333,9 @@ private[history] class FsHistoryProvider(conf: SparkConf) extends ApplicationHis ...@@ -333,8 +333,9 @@ private[history] class FsHistoryProvider(conf: SparkConf) extends ApplicationHis
} }
try { try {
val appListener = new ApplicationEventListener val appListener = new ApplicationEventListener
val appCompleted = isApplicationCompleted(eventLog)
bus.addListener(appListener) bus.addListener(appListener)
bus.replay(logInput, logPath.toString) bus.replay(logInput, logPath.toString, !appCompleted)
new FsApplicationHistoryInfo( new FsApplicationHistoryInfo(
logPath.getName(), logPath.getName(),
appListener.appId.getOrElse(logPath.getName()), appListener.appId.getOrElse(logPath.getName()),
...@@ -343,7 +344,7 @@ private[history] class FsHistoryProvider(conf: SparkConf) extends ApplicationHis ...@@ -343,7 +344,7 @@ private[history] class FsHistoryProvider(conf: SparkConf) extends ApplicationHis
appListener.endTime.getOrElse(-1L), appListener.endTime.getOrElse(-1L),
getModificationTime(eventLog).get, getModificationTime(eventLog).get,
appListener.sparkUser.getOrElse(NOT_STARTED), appListener.sparkUser.getOrElse(NOT_STARTED),
isApplicationCompleted(eventLog)) appCompleted)
} finally { } finally {
logInput.close() logInput.close()
} }
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment