[SPARK-15951] Change Executors Page to use datatables to support sorting columns and searching #13670

kishorvpatil · 2016-06-14T21:19:02Z

Create the executorspage-template.html for displaying application information in datables.
Added REST API endpoint "allexecutors" to be able to see all executors created for particular job.
The executorspage.js uses jQuery to access the data from /api/v1/applications/appid/allexecutors REST API, and use DataTable to display executors for the application. It also, generates summary of dead/live and total executors created during life of the application.
Similar changes applicable to Executors Page on history server for a given application.

Snapshots for how it looks like now:

New Executors Page screenshot looks like this:

ajbozarth · 2016-06-14T23:55:46Z

Can you post a screenshot of how it looks after the change? I'll take a look through the code when I have a chance. Also @zhuoliu want to take a look since you were the one who added JQuery DataTables to the History Server

kishorvpatil · 2016-06-15T15:22:12Z

@ajbozarth , Thank you for loooking into this patch. I have added screenshot with annotation. I hope that helps.

tgravescs · 2016-06-15T18:33:01Z

Jenkins, ok to test

SparkQA · 2016-06-15T20:23:17Z

Test build #60584 has finished for PR 13670 at commit 56f3901.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

tgravescs · 2016-06-15T20:44:19Z

Jenkins, test this please

SparkQA · 2016-06-15T22:37:16Z

Test build #60601 has finished for PR 13670 at commit 56f3901.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

ajbozarth · 2016-06-15T22:43:31Z

Jenkins, test this please

SparkQA · 2016-06-16T00:53:02Z

Test build #60612 has finished for PR 13670 at commit 56f3901.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

zhuoliu · 2016-06-16T16:58:18Z

Generally looks great to me, I am good with it as long as @tgravescs verifies. One minor concern is that we may move the utility functions from executorspage.js and historypage.ps to a common location.

ajbozarth · 2016-06-16T18:10:48Z

My comment didn't post last night:
I haven't finished reading through your code yet, but after checking it out and starting up the web ui I get the following error on the executors page.

ajbozarth · 2016-06-16T20:53:22Z

I agree with @zhuoliu we should move common/util functions into a datatables util js.

For the above error, I pinpointed that it is baseURI that is null. I also never see the "Loading Executors Page" message i see in the code. I have also checked that the page doesn't load in any browser. @zhuoliu have you tried checking out and running the code?

SparkQA · 2016-06-22T23:56:46Z

Test build #61070 has finished for PR 13670 at commit 3b4962a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-06-28T16:23:19Z

Test build #61387 has started for PR 13670 at commit c73e905.

kishorvpatil · 2016-06-28T20:18:26Z

@zhuoliu , i have removed unwanted, unused javascript functions in the executorspage.js. There are no duplicate functions now between historyserver.js and this script..

kishorvpatil · 2016-06-28T20:19:59Z

@ajbozarth, I do not see how I could produce the exception you see. Please help me reproduce this so that I can address it.

ajbozarth · 2016-06-28T20:23:48Z

docs/monitoring.md

+    <td>A list of all active executors for the given application.</td>
+  </tr>
+  <tr>
+    <td><code>/applications/[app-id]/executors</code></td>


These are duplicate end points: /applications/[app-id]/executors

yep, I think you meant allexecutors

ajbozarth · 2016-06-28T20:24:27Z

I'll re-checkout later today and see if I can reproduce the error for you

ajbozarth · 2016-06-28T23:50:44Z

I get the error by just starting up a spark-shell, but I only get it on Safari, it works fine on Chrome and Firefox. I'll make a line note of what I see going wrong

ajbozarth · 2016-06-28T23:52:58Z

core/src/main/resources/org/apache/spark/ui/static/executorspage.js

+
+function createRESTEndPoint() {
+    var parser = document.createElement('a');
+    var words = parser.baseURI.split('/');


In Safari baseURI is null, it seems to be how Safari handles document.createElement, you may want to research it and possibly find another way to get the URL, there are no other uses of baseURI in the Spark code base.

As mentioned, I switched it to use document.baseURI which works with Safari as well.

kishorvpatil · 2016-07-14T20:39:11Z

@tgravescs , moved formatBytes to utils.js.

SparkQA · 2016-07-14T22:07:30Z

Test build #62343 has finished for PR 13670 at commit bc132b1.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-07-14T22:55:16Z

Test build #62346 has finished for PR 13670 at commit 66df19f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

tgravescs · 2016-07-15T19:56:03Z

@ajbozarth were you able to try the latest version?

tgravescs · 2016-07-15T21:11:03Z

core/src/main/resources/org/apache/spark/ui/static/executorspage.js

+    });
+
+    executorsSummary = $("#active-executors");
+    searchString = executorsSummary["context"]["location"]["search"];


unused, remove

ajbozarth · 2016-07-16T02:26:51Z

Been busy the last couple days, I'll take a look at this Monday

kishorvpatil · 2016-07-18T15:14:46Z

@tgravescs, I removed unwanted code/comment from the javascript.

SparkQA · 2016-07-18T17:14:57Z

Test build #62472 has finished for PR 13670 at commit e81d45b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

ajbozarth · 2016-07-19T00:02:19Z

Sorry I couldn't get to this today, I'll do my best to take a look in the morning

ajbozarth · 2016-07-19T23:52:55Z

core/src/main/resources/org/apache/spark/ui/static/executorspage.js

+    return activeTasks > 0 ? ("hsla(240, 100%, 50%, " + activeTasksAlpha(activeTasks, maxTasks) + ")") : "";
+}
+
+function activeTasksColor(activeTasks, maxTasks) {


This is never used

ajbozarth · 2016-07-19T23:57:17Z

I checked out out the latest code and ran it and it looks good, I also looked through the code and outside of the two unused functions I found the code looks good too.

Im comfortable giving this a LGTM

We may want to start setting a standard way to convert pages to use DataTables, this seemed more a difficult process than I thought it would be

kishorvpatil · 2016-07-20T14:50:24Z

@ajbozarth Thank you for review. I removed unused methods from the the javascript.

I agree converting pages to DataTables is tricky and listing some standard way might be a huge help.

SparkQA · 2016-07-20T17:05:40Z

Test build #62607 has finished for PR 13670 at commit 87301c1.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

tgravescs · 2016-07-20T17:21:38Z

+1, thanks @kishorvpatil @ajbozarth

nblintao · 2016-07-21T14:35:51Z

Hi, @kishorvpatil. Cool improvement! I'm developing the Executors Page at #14204 based on your changes. But I have a small question about your new implementation of the tables.
In the implementation of ExecutorsPage.scala before, logsExist is checked and there will be no Logs column if it's empty for all rows. But now, I haven't found similar implementation in the executorspage.js. I'm not sure whether it is handled by JQuery DataTables automatically, or I'm missing something, or this is not available for now? Thanks.

kishorvpatil · 2016-07-21T16:50:43Z

@nblintao , I am not sure why we had the check of existence of logs - specially when the reference to logs is static link for stdout and stderr.
Unless there is scenario where there could be multiple log entries and dynamic list of logs- i suggest that would require coding of different solution - mere check for existence and dropping out column altogether would not help.

ajbozarth · 2016-07-21T17:01:10Z

@kishorvpatil I can't believe I missed that. The logs only exist if you have logging turned on in your confs, if it's off there won't be any logs and no reason for the column. @nblintao if you want to file a JIRA either you @kishorvpatil or I can quickly fix that

tgravescs · 2016-07-21T18:29:21Z

personally I don't see this as a big deal. As long as it doesn't show the links, the column being there doesn't hurt anything and it keeps things more consistent. Perhaps change it to say Not Available or something?

If you thinkts its an issue lets file a separate jira for this and discuss further there.

ajbozarth · 2016-07-21T21:13:46Z

I opened SPARK-16673 for the issue since it also apply to the Thread Dump column which the History Server shouldn't display.

…and Thread Dump columns ## What changes were proposed in this pull request? When #13670 switched `ExecutorsPage` to use JQuery DataTables it incidentally removed the conditional for the Logs and Thread Dump columns. I reimplemented the conditional display of the Logs and Thread dump columns as it was before the switch. ## How was this patch tested? Manually tested and dev/run-tests ![both](https://cloud.githubusercontent.com/assets/13952758/17186879/da8dd1a8-53eb-11e6-8b0c-d0ff0156a9a7.png) ![dump](https://cloud.githubusercontent.com/assets/13952758/17186881/dab08a04-53eb-11e6-8b1c-50ffd0bf2ae8.png) ![logs](https://cloud.githubusercontent.com/assets/13952758/17186880/dab04d00-53eb-11e6-8754-68dd64d6d9f4.png) Author: Alex Bozarth <[email protected]> Closes #14382 from ajbozarth/spark16673.

merlintang · 2016-12-07T00:14:16Z

@kishorvpatil
you provided the function allexecutors, which is used to return the dead and active executor information.

For the document
http://spark.apache.org/docs/latest/monitoring.html
for
/applications/[app-id]/executors A list of all executors for the given application.

We had better document more clearly what is meaning of functions for the 2.1 version.
/applications/[app-id]/executors xxx
/applications/[app-id]/allexecutors xxxx

This confused people, because our test already run into this issue.

kishorvpatil added 4 commits June 17, 2016 16:23

Add AllExecutors REST Endpoint

f66f2ea

Adding ExecutorsPage template

8df49af

History page for Executors use mustache template

c0e2afa

Merge branch 'master' of git://git.apache.org/spark into SPARK-15951

3b4962a

kishorvpatil force-pushed the execTemplates branch from 56f3901 to 3b4962a Compare June 22, 2016 21:55

Remove unused javascript methods

c73e905

ajbozarth reviewed Jun 28, 2016
View reviewed changes

tgravescs reviewed Jul 15, 2016
View reviewed changes

Remove commented unused js lines

e81d45b

ajbozarth reviewed Jul 19, 2016
View reviewed changes

Remove unused javascript functions

87301c1

asfgit closed this in b9bab4d Jul 20, 2016

nblintao mentioned this pull request Jul 21, 2016

[SPARK-16520] [WEBUI] Link executors to corresponding worker pages #14204

Closed

ajbozarth mentioned this pull request Jul 27, 2016

[SPARK-16673] [Web UI] New Executor Page removed conditional for Logs and Thread Dump columns #14382

Closed

panbingkun mentioned this pull request May 26, 2023

[SPARK-43576][CORE] Remove unused declarations from Core module #41218

Closed

[SPARK-15951] Change Executors Page to use datatables to support sorting columns and searching #13670

[SPARK-15951] Change Executors Page to use datatables to support sorting columns and searching #13670

Uh oh!

Conversation

kishorvpatil commented Jun 14, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ajbozarth commented Jun 14, 2016

Uh oh!

kishorvpatil commented Jun 15, 2016

Uh oh!

tgravescs commented Jun 15, 2016

Uh oh!

SparkQA commented Jun 15, 2016

Uh oh!

tgravescs commented Jun 15, 2016

Uh oh!

SparkQA commented Jun 15, 2016

Uh oh!

ajbozarth commented Jun 15, 2016

Uh oh!

SparkQA commented Jun 16, 2016

Uh oh!

zhuoliu commented Jun 16, 2016

Uh oh!

ajbozarth commented Jun 16, 2016

Uh oh!

ajbozarth commented Jun 16, 2016

Uh oh!

SparkQA commented Jun 22, 2016

Uh oh!

SparkQA commented Jun 28, 2016

Uh oh!

kishorvpatil commented Jun 28, 2016

Uh oh!

kishorvpatil commented Jun 28, 2016

Uh oh!

ajbozarth Jun 28, 2016

Choose a reason for hiding this comment

Uh oh!

tgravescs Jul 12, 2016

Choose a reason for hiding this comment

Uh oh!

kishorvpatil Jul 13, 2016

Choose a reason for hiding this comment

Uh oh!

ajbozarth commented Jun 28, 2016

Uh oh!

ajbozarth commented Jun 28, 2016

Uh oh!

ajbozarth Jun 28, 2016

Choose a reason for hiding this comment

Uh oh!

kishorvpatil Jul 13, 2016

Choose a reason for hiding this comment

Uh oh!

kishorvpatil commented Jul 14, 2016

Uh oh!

SparkQA commented Jul 14, 2016

Uh oh!

SparkQA commented Jul 14, 2016

Uh oh!

tgravescs commented Jul 15, 2016

Uh oh!

tgravescs Jul 15, 2016

Choose a reason for hiding this comment

Uh oh!

ajbozarth commented Jul 16, 2016

Uh oh!

kishorvpatil commented Jul 18, 2016

Uh oh!

SparkQA commented Jul 18, 2016

Uh oh!

ajbozarth commented Jul 19, 2016

Uh oh!

ajbozarth Jul 19, 2016

Choose a reason for hiding this comment

Uh oh!

ajbozarth commented Jul 19, 2016

Uh oh!

kishorvpatil commented Jun 14, 2016 •

edited

Loading