-
- Downloads
[SPARK-4913] Fix incorrect event log path
SPARK-2261 uses a single file to log events for an app. `eventLogDir` in `ApplicationDescription` is replaced with `eventLogFile`. However, `ApplicationDescription` in `SparkDeploySchedulerBackend` is initialized with `SparkContext`'s `eventLogDir`. It is just the log directory, not the actual log file path. `Master.rebuildSparkUI` can not correctly rebuild a new SparkUI for the app. Because the `ApplicationDescription` is remotely registered with `Master` and the app's id is then generated in `Master`, we can not get the app id in advance before registration. So the received description needs to be modified with correct `eventLogFile` value. Author: Liang-Chi Hsieh <viirya@gmail.com> Closes #3755 from viirya/fix_app_logdir and squashes the following commits: 5e0ea35 [Liang-Chi Hsieh] Revision for comment. b5730a1 [Liang-Chi Hsieh] Fix incorrect event log path. Closes #3777 (a duplicate PR for the same JIRA)
Showing
- core/src/main/scala/org/apache/spark/deploy/ApplicationDescription.scala 1 addition, 1 deletion...cala/org/apache/spark/deploy/ApplicationDescription.scala
- core/src/main/scala/org/apache/spark/deploy/master/Master.scala 16 additions, 4 deletions...rc/main/scala/org/apache/spark/deploy/master/Master.scala
Loading
Please register or sign in to comment