[SPARK-16809] enable history server links in dispatcher UI

[mgummelt]

What changes were proposed in this pull request?

Links the Spark Mesos Dispatcher UI to the history server UI

• adds spark.mesos.dispatcher.historyServer.url
• explicitly generates frameworkIDs for the launched drivers, so the dispatcher knows how to correlate drivers and frameworkIDs

How was this patch tested?

manual testing

[mgummelt]

cc @skonto. This somewhat relates to your work on https://issues.apache.org/jira/browse/SPARK-13401, but it seems to be disjoint.

[SparkQA]

Test build #63027 has finished for PR 14414 at commit 6e97ec1.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[jaceklaskowski]

Would it be possible to define a constant for "spark.mesos.dispatcher.historyServer.url" (as is in SQL and YARN modules)? See https://github.com/apache/spark/blob/master/yarn/src/main/scala/org/apache/spark/deploy/yarn/config.scala.

[mgummelt]

Ah, I wasn't aware of that class. We should definitely use it, but I'd like to make that change consistently. I made a JIRA for the migration: https://issues.apache.org/jira/browse/SPARK-16881

[mgummelt]

@jaceklaskowski PTAL

[SparkQA]

Test build #63180 has finished for PR 14414 at commit b36eb1d.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[skonto]

You need to add UI screenshots for the changes at the description.

[skonto]

cc @skonto. This somewhat relates to your work on https://issues.apache.org/jira/browse/SPARK-13401, but it seems to be disjoint.

The link is not correct i guess you meant https://issues.apache.org/jira/browse/SPARK-13041 ;)

[skonto]

FinishDate is missing.

[mgummelt]

fixed

[skonto]

I will add more comment im trying to test it.

[skonto]

I tested it with our suite. Two comments:
a. Its not convenient to have only one option to add the history server uri. I had to create a properties file to pass it to the dispatcher process. I would expect to pass it with --conf

b. From what I see from here:



when you have one driver starting and stopping contexts then history link can only point to the very first context. I guess that is a limitation.

[mgummelt]

a. Its not convenient to have only one option to add the history server uri. I had to create a properties file to pass it to the dispatcher process. I would expect to pass it with --conf

I agree, and we should change this at some point, but that's not the goal of this PR.

when you have one driver starting and stopping contexts then history link can only point to the very first context. I guess that is a limitation.

Yea, it's a limitation. The dispatcher knows about drivers, but the history server knows about apps (i.e. contexts), and this relationship is one to many. This is a simple way to connect the two in the common case of 1 driver having 1 context. To solve the 1-to-many case, we'd need some way of the driver communicating back to the dispatcher for every context that it creates. That's a more difficult problem, and I don't think it's worth solving right now.

[SparkQA]

Test build #63382 has finished for PR 14414 at commit 263aaa4.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[skonto]

I agree just to have a reminder for what we can achieve. LGTM other than that.

[mgummelt]

@srowen Can you merge?

[srowen]

Merged to master