Skip to content
Snippets Groups Projects
Commit c871e2da authored by moussa taifi's avatar moussa taifi Committed by Andrew Or
Browse files

Add a note for context termination for History server on Yarn

The history server on Yarn only shows completed jobs. This adds a note concerning the needed explicit context termination at the end of a spark job which is a best practice anyway.
Related to SPARK-2972 and SPARK-3458

Author: moussa taifi <moutai10@gmail.com>

Closes #4721 from moutai/add-history-server-note-for-closing-the-spark-context and squashes the following commits:

9f5b6c3 [moussa taifi] Fix upper case typo for YARN
3ad3db4 [moussa taifi] Add context termination for History server on Yarn
parent 3fb53c02
No related branches found
No related tags found
No related merge requests found
...@@ -173,6 +173,8 @@ follows: ...@@ -173,6 +173,8 @@ follows:
Note that in all of these UIs, the tables are sortable by clicking their headers, Note that in all of these UIs, the tables are sortable by clicking their headers,
making it easy to identify slow tasks, data skew, etc. making it easy to identify slow tasks, data skew, etc.
Note that the history server only displays completed Spark jobs. One way to signal the completion of a Spark job is to stop the Spark Context explicitly (`sc.stop()`), or in Python using the `with SparkContext() as sc:` to handle the Spark Context setup and tear down, and still show the job history on the UI.
# Metrics # Metrics
Spark has a configurable metrics system based on the Spark has a configurable metrics system based on the
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment