[SPARK-15590] [WEBUI] Paginate Job Table in Jobs tab #13620

nblintao · 2016-06-11T17:37:25Z

What changes were proposed in this pull request?

This patch adds pagination support for the Job Tables in the Jobs tab. Pagination is provided for all of the three Job Tables (active, completed, and failed). Interactions (jumping, sorting, and setting page size) for paged tables are also included.

The diff didn't keep track of some lines based on the original ones. The function makeRowof the original AllJobsPage.scala is reused. They are separated at the beginning of the function jobRow (L427-439) and the function row(L594-618) in the new AllJobsPage.scala.

How was this patch tested?

Tested manually by using checking the Web UI after completing and failing hundreds of jobs.
Generate completed jobs by:

val d = sc.parallelize(Array(1,2,3,4,5))
for(i <- 1 to 255){ var b = d.collect() }

Generate failed jobs by calling the following code multiple times:

var b = d.map(_/0).collect()

Interactions like jumping, sorting, and setting page size are all tested.

This shows the pagination for completed jobs:

This shows the sorting works in job tables:

This shows the pagination for failed jobs and the effect of jumping and setting page size:

andrewor14 · 2016-06-13T23:11:04Z

add to whitelist

andrewor14 · 2016-06-13T23:13:57Z

core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala

+    r
+  }
+
+  def slicedJobIds: Set[Int] = _slicedJobIds


used anywhere?

Sorry, it's deprecated. I will remove it. Thanks.

andrewor14 · 2016-06-13T23:16:00Z

@nblintao this looks pretty good. There were a few methods that don't seem to be used anywhere. Am I missing something?

SparkQA · 2016-06-14T01:05:09Z

Test build #60444 has finished for PR 13620 at commit 4db0e09.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-06-14T07:18:39Z

Test build #60469 has finished for PR 13620 at commit 6682e86.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-06-15T03:38:47Z

Test build #60543 has finished for PR 13620 at commit 41bb189.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

ajbozarth · 2016-06-15T22:13:39Z

This looks great, I checked out the code and it looks good locally on my machine as well. I've read through the code and it looks good with the fixes @andrewor14 suggested so ~~LGTM~~ (see below)

andrewor14 · 2016-06-16T17:34:08Z

also cc @zsxwing

ajbozarth · 2016-06-16T23:44:11Z

Just opened this in the history server to double check the bug I found on your stages pr and discovered that this is completely non-functional on the history server, so I'm striking my earlier ~~LGTM~~

nblintao · 2016-06-17T02:01:18Z

@ajbozarth Thanks for your checking and reporting. I didn't notice that the UI for history servers are different and just changed the AllJobsPage.

I will try to check the page for history servers. But do you think it would be better if we use another PR to solve this (adding the pagination support for the jobs page in history servers)?

ajbozarth · 2016-06-17T02:09:45Z

They actually share the core majority of their code so your pagination is already there, it just can't change pages or change the number of shown jobs. I think its has to do with how your calling and parsing the http requests since I didn't see any parameters in the url when I would try to change pages

nblintao · 2016-06-17T02:21:41Z

@ajbozarth Got it, that's a really fatal bug. I will fix it in this PR. Thanks again!

nblintao · 2016-06-23T17:12:48Z

This PR has known bugs about linking. I have fixed the related bugs in #13708. In order to avoid conflicts, I will not update this PR until #13708 is merged.
So, please do NOT review or test this PR for now. It would be nice if you could help review or test #13708. Thanks!

## What changes were proposed in this pull request? This patch adds pagination support for the Stage Tables in the Stage tab. Pagination is provided for all of the four Job Tables (active, pending, completed, and failed). Besides, the paged stage tables are also used in JobPage (the detail page for one job) and PoolPage. Interactions (jumping, sorting, and setting page size) for paged tables are also included. ## How was this patch tested? Tested manually by using checking the Web UI after completing and failing hundreds of jobs. Same as the testings for [Paginate Job Table in Jobs tab](apache#13620). This shows the pagination for completed stages: ![paged stage table](https://cloud.githubusercontent.com/assets/5558370/16125696/5804e35e-3427-11e6-8923-5c5948982648.png) Author: Tao Lin <[email protected]> Closes apache#13708 from nblintao/stageTable.

SparkQA · 2016-07-07T14:02:41Z

Test build #61910 has finished for PR 13620 at commit 649bb19.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

nblintao · 2016-07-07T14:26:56Z

I believe this commit has resolved the bugs reported by @ajbozarth. It looks well on history server pages now, and it could keep the status of other tables while changing one.
Could you please help test or review it if you are available? Thanks! @andrewor14 @zsxwing @ajbozarth

ajbozarth · 2016-07-11T21:48:40Z

@nblintao I just finished testing your changes and it looks good. I did find a odd behavior though, but I think it's intentional in the original pagination code. If you make the table large enough to fit all the jobs the pagination toolbar disappears. Given you can just click the back button to fix it I don't think it's an issue to be fixed, but it is a bit annoying. Another fyi is that only ~1000 jobs are shown still, but I'm assuming the pagination doesn't fix the reasoning behind only showing the most recent 1000 jobs.

tl;dr LGTM

yhuai · 2016-07-24T22:59:50Z

core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala

+      jobs: Seq[JobUIData]): Seq[Node] = {
+    val allParameters = request.getParameterMap.asScala.toMap
+    val parameterOtherTable = allParameters.filterNot(_._1.startsWith(jobTag))
+      .map(para => para._1 + "=" + para._2(0))


What does parameterOtherTable do?

It keeps the status of other tables on the page. See #13708 (comment)

yhuai · 2016-07-24T23:08:12Z

@nblintao Can you comment on your PR to explain which parts are new code and which parts are based on existing code (comments like https://github.com/apache/spark/pull/13620/files#r71997586)?

yhuai · 2016-07-24T23:08:41Z

core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala

+  override def row(jobTableRow: JobTableRowData): Seq[Node] = {
+    val job = jobTableRow.jobData
+
+    <tr id={"job-" + job.jobId}>


Copied from https://github.com/apache/spark/pull/13620/files#diff-9a486b2f4e30d5375c3541571aec616cL240?

nblintao · 2016-07-25T10:25:05Z

@yhuai I updated the description of the PR:

The diff didn't keep track of some lines based on the original ones. The function makeRowof the original AllJobsPage.scala is reused. They are separated at the beginning of the function jobRow (L427-439) and the function row(L594-618) in the new AllJobsPage.scala.

And thanks for your review!

zsxwing · 2016-07-25T22:12:50Z

retest this please

zsxwing · 2016-07-25T22:13:21Z

LGTM. Let's run the test again since this is a pretty old PR. Thanks!

SparkQA · 2016-07-26T00:29:10Z

Test build #62846 has finished for PR 13620 at commit 649bb19.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

zsxwing · 2016-07-26T00:35:29Z

Merging to master.

nblintao · 2016-07-26T01:05:00Z

Great! Thanks, @andrewor14 @ajbozarth @yhuai @zsxwing.

## What changes were proposed in this pull request? This issue was reported by wangyum. In the AllJobsPage, JobPage and StagePage, the description length was limited before like as follows. ![ui-2 0 0](https://cloud.githubusercontent.com/assets/4736016/21319673/8b225246-c651-11e6-9041-4fcdd04f4dec.gif) But recently, the limitation seems to have been accidentally removed. ![ui-2 1 0](https://cloud.githubusercontent.com/assets/4736016/21319825/104779f6-c652-11e6-8bfa-dfd800396352.gif) The cause is that some tables are no longer `sortable` class although they were, and `sortable` class does not only mark tables as sortable but also limited the width of their child `td` elements. The reason why now some tables are not `sortable` class is because another sortable mechanism was introduced by #13620 and #13708 with pagination feature. To fix this issue, I've introduced new class `table-cell-width-limited` which limits the description cell width and the description is like what it was. <img width="1260" alt="2016-12-20 1 00 34" src="https://cloud.githubusercontent.com/assets/4736016/21320478/89141c7a-c654-11e6-8494-f8f91325980b.png"> ## How was this patch tested? Tested manually with my browser. Author: Kousuke Saruta <[email protected]> Closes #16338 from sarutak/SPARK-18837.

## What changes were proposed in this pull request? This issue was reported by wangyum. In the AllJobsPage, JobPage and StagePage, the description length was limited before like as follows. ![ui-2 0 0](https://cloud.githubusercontent.com/assets/4736016/21319673/8b225246-c651-11e6-9041-4fcdd04f4dec.gif) But recently, the limitation seems to have been accidentally removed. ![ui-2 1 0](https://cloud.githubusercontent.com/assets/4736016/21319825/104779f6-c652-11e6-8bfa-dfd800396352.gif) The cause is that some tables are no longer `sortable` class although they were, and `sortable` class does not only mark tables as sortable but also limited the width of their child `td` elements. The reason why now some tables are not `sortable` class is because another sortable mechanism was introduced by #13620 and #13708 with pagination feature. To fix this issue, I've introduced new class `table-cell-width-limited` which limits the description cell width and the description is like what it was. <img width="1260" alt="2016-12-20 1 00 34" src="https://cloud.githubusercontent.com/assets/4736016/21320478/89141c7a-c654-11e6-8494-f8f91325980b.png"> ## How was this patch tested? Tested manually with my browser. Author: Kousuke Saruta <[email protected]> Closes #16338 from sarutak/SPARK-18837. (cherry picked from commit f2ceb2a) Signed-off-by: Sean Owen <[email protected]>

## What changes were proposed in this pull request? This issue was reported by wangyum. In the AllJobsPage, JobPage and StagePage, the description length was limited before like as follows. ![ui-2 0 0](https://cloud.githubusercontent.com/assets/4736016/21319673/8b225246-c651-11e6-9041-4fcdd04f4dec.gif) But recently, the limitation seems to have been accidentally removed. ![ui-2 1 0](https://cloud.githubusercontent.com/assets/4736016/21319825/104779f6-c652-11e6-8bfa-dfd800396352.gif) The cause is that some tables are no longer `sortable` class although they were, and `sortable` class does not only mark tables as sortable but also limited the width of their child `td` elements. The reason why now some tables are not `sortable` class is because another sortable mechanism was introduced by apache#13620 and apache#13708 with pagination feature. To fix this issue, I've introduced new class `table-cell-width-limited` which limits the description cell width and the description is like what it was. <img width="1260" alt="2016-12-20 1 00 34" src="https://cloud.githubusercontent.com/assets/4736016/21320478/89141c7a-c654-11e6-8494-f8f91325980b.png"> ## How was this patch tested? Tested manually with my browser. Author: Kousuke Saruta <[email protected]> Closes apache#16338 from sarutak/SPARK-18837.

paginate job tables

4db0e09

andrewor14 reviewed Jun 13, 2016
View reviewed changes

fix default order & fix for job groups & rm unneccessary def

6682e86

add notions of override

41bb189

nblintao mentioned this pull request Jun 16, 2016

[SPARK-15591] [WEBUI] Paginate Stage Table in Stages tab #13708

Closed

fix link problems

649bb19

yhuai reviewed Jul 24, 2016
View reviewed changes

asfgit closed this in db36e1e Jul 26, 2016

sarutak mentioned this pull request Dec 19, 2016

[SPARK-18837][WEBUI] Very long stage descriptions do not wrap in the UI #16338

Closed

[SPARK-15590] [WEBUI] Paginate Job Table in Jobs tab #13620

[SPARK-15590] [WEBUI] Paginate Job Table in Jobs tab #13620

Uh oh!

Conversation

nblintao commented Jun 11, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

andrewor14 commented Jun 13, 2016

Uh oh!

andrewor14 Jun 13, 2016

Choose a reason for hiding this comment

Uh oh!

nblintao Jun 14, 2016

Choose a reason for hiding this comment

Uh oh!

andrewor14 commented Jun 13, 2016

Uh oh!

SparkQA commented Jun 14, 2016

Uh oh!

SparkQA commented Jun 14, 2016

Uh oh!

SparkQA commented Jun 15, 2016

Uh oh!

ajbozarth commented Jun 15, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

andrewor14 commented Jun 16, 2016

Uh oh!

ajbozarth commented Jun 16, 2016

Uh oh!

nblintao commented Jun 17, 2016

Uh oh!

ajbozarth commented Jun 17, 2016

Uh oh!

nblintao commented Jun 17, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nblintao commented Jun 23, 2016

Uh oh!

SparkQA commented Jul 7, 2016

Uh oh!

nblintao commented Jul 7, 2016

Uh oh!

ajbozarth commented Jul 11, 2016

Uh oh!

yhuai Jul 24, 2016

Choose a reason for hiding this comment

Uh oh!

nblintao Jul 25, 2016

Choose a reason for hiding this comment

Uh oh!

yhuai commented Jul 24, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yhuai Jul 24, 2016

Choose a reason for hiding this comment

Uh oh!

nblintao Jul 25, 2016

Choose a reason for hiding this comment

Uh oh!

nblintao commented Jul 25, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zsxwing commented Jul 25, 2016

Uh oh!

zsxwing commented Jul 25, 2016

Uh oh!

SparkQA commented Jul 26, 2016

Uh oh!

zsxwing commented Jul 26, 2016

Uh oh!

nblintao commented Jul 26, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

nblintao commented Jun 11, 2016 •

edited

Loading

ajbozarth commented Jun 15, 2016 •

edited

Loading

nblintao commented Jun 17, 2016 •

edited

Loading

yhuai commented Jul 24, 2016 •

edited

Loading

nblintao commented Jul 25, 2016 •

edited

Loading