Skip to content

Commit 14e042b

Browse files
moutaiAndrew Or
authored andcommitted
Add a note for context termination for History server on Yarn
The history server on Yarn only shows completed jobs. This adds a note concerning the needed explicit context termination at the end of a spark job which is a best practice anyway. Related to SPARK-2972 and SPARK-3458 Author: moussa taifi <[email protected]> Closes #4721 from moutai/add-history-server-note-for-closing-the-spark-context and squashes the following commits: 9f5b6c3 [moussa taifi] Fix upper case typo for YARN 3ad3db4 [moussa taifi] Add context termination for History server on Yarn (cherry picked from commit c871e2d) Signed-off-by: Andrew Or <[email protected]>
1 parent f74bccb commit 14e042b

File tree

1 file changed

+2
-0
lines changed

1 file changed

+2
-0
lines changed

docs/monitoring.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -133,6 +133,8 @@ represents an application's event logs. This creates a web interface at
133133
Note that in all of these UIs, the tables are sortable by clicking their headers,
134134
making it easy to identify slow tasks, data skew, etc.
135135

136+
Note that the history server only displays completed Spark jobs. One way to signal the completion of a Spark job is to stop the Spark Context explicitly (`sc.stop()`), or in Python using the `with SparkContext() as sc:` to handle the Spark Context setup and tear down, and still show the job history on the UI.
137+
136138
# Metrics
137139

138140
Spark has a configurable metrics system based on the

0 commit comments

Comments
 (0)