[SPARK-31971][WEBUI] Add pagination support for all jobs timeline #28803

sarutak · 2020-06-11T20:51:54Z

What changes were proposed in this pull request?

This PR proposes to add pagination support for all-jobs timeline.

Why are the changes needed?

If there are lots of jobs, rendering performance of all jobs timeline can significantly goes down. This issue is reported in SPARK-31967.
For example, after the following operation, UI loading can take >40 sec.

(1 to 1000).foreach(_ => sc.parallelize(1 to 10).collect)

Although it's not the fundamental solution, pagination can mitigate the issue.

Does this PR introduce any user-facing change?

Yes.
Before this change, all displayable jobs are displayed in one page.

After this change, 2 text boxes are added.
One is to set the maximum number of jobs to be displayed in a page and the other is to set the page number.
Go button is also added and if it's pushed, jobs are re-rendered based on the page size and page number.

How was this patch tested?

Additional testcase with the following command.

build/mvn -Dguava.version=25.0-jre -Dspark.test.webdriver.chrome.driver=/path/to/chromedriver -Dtest.default.exclude.tags=  -Dtest=none -DwildcardSuites="org.apache.spark.ui.ChromeUISeleniumSuite" test

sarutak · 2020-06-11T20:54:09Z

cc: @gengliangwang
This improvement is not the fundamental solution for SPARK-31967 but mitigate it so I've filed as an independent issue.

gengliangwang · 2020-06-11T21:40:23Z

@sarutak Thanks for the quick fix. I try on my local setup and it works.

gengliangwang · 2020-06-11T21:40:44Z

I think we should make the two pagination sections consistent

SparkQA · 2020-06-11T23:25:52Z

Test build #123864 has finished for PR 28803 at commit 90b132b.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

gengliangwang · 2020-06-12T03:21:25Z

@sarutak I just confirmed that the issue is caused by #28192 . The issus can't be reproduced before the change.

sarutak · 2020-06-12T04:26:16Z

@gengliangwang Yes, I've noticed that. The performance issue of vis-timeline happens for newer version of vis.js.

gengliangwang · 2020-06-12T04:32:40Z

So, I am actually -1 in this solution, for two reasons:

if there are many stages in one job, the perf issue still exists
the two paginations in one page seem weird.

I tried a different version 4.21.0 from https://cdnjs.com/libraries/vis
The infinite drawing issue seems better if the zoom is not enabled.
How about we changing to that version? or, just reverting #28192?

gengliangwang · 2020-06-12T04:36:30Z

Let me create a PR for downgrading to 4.21.0 now.

sarutak · 2020-06-12T04:51:55Z

if there are many stages in one job, the perf issue still exists

the two paginations in one page seem weird.

For 1), AllJobsPage is not affected even if a job has many stages isn't it? JobPage can be affect but we can apply the same solution. For 2), Yes, it's a little bit weird but the similar solution already exists in StagePage.

Even if no performance issue exists, lots of items rendered in one page can be too noisy.

gengliangwang · 2020-06-12T05:06:19Z

@sarutak I just created #28806.
I believe downgrading the vis version is a safer solution. We need to hotfix this to Spark 3.0 ASAP

gengliangwang · 2020-06-12T05:18:59Z

Even if no performance issue exists, lots of items rendered in one page can be too noisy.

OK, how about we merge #28806 to 3.0/2.4 first.
And start the pagination feature from 3.1 if you insist?

sarutak · 2020-06-12T05:38:21Z

OK, how about we merge #28806 to 3.0/2.4 first.
And start the pagination feature from 3.1 if you insist?

I agree. I'll check that PR soon.

…ination-support-for-all-jobs-timeline

sarutak · 2020-06-13T06:08:37Z

I think we should make the two pagination sections consistent

The upper text fields are for timeline, while the lower ones are for the table so I think they should be independent each other.

SparkQA · 2020-06-13T07:05:02Z

Test build #123965 has finished for PR 28803 at commit 0270bb9.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

sarutak · 2020-06-13T07:10:20Z

retest this please.

SparkQA · 2020-06-13T09:47:01Z

Test build #123969 has finished for PR 28803 at commit 0270bb9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

sarutak · 2020-07-01T02:43:51Z

Hi @gengliangwang, shall we restart discussion?

sarutak · 2020-07-08T03:30:29Z

cc: @dongjoon-hyun

gengliangwang · 2020-07-08T04:38:26Z

retest this please

gengliangwang · 2020-07-08T04:46:50Z

core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala

+                    method="get"
+                    action=""
+                    class="form-inline justify-content-end"
+                    style="width: 50%; margin-left: auto; margin-bottom: 0px;">


nit: shall we move the style to webui.css?

Thanks. I'll consider it.

gengliangwang · 2020-07-08T04:53:04Z

core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala

+                  Pages. Jump to</label>
+                <input type="text"
+                       name="jobs.eventTimelinePageNumber"
+                       id={s"form-event-timeline-page-no"}


Nit: I find similar code in StagePage.scala. But why do we need {s".."} here?

Exactly. I'll remove s prefix.

gengliangwang · 2020-07-08T04:56:13Z

@sarutak I am really sorry I missed your ping last week.
The perf problem is resolved in #28806. If we want to avoid perf issue in the long term, I think we can avoid rendering the timeline by default.
I am +0 with the proposal. The pagination makes the visualization simpler, but also makes the UI usage more complex. If you insist, let's merge this one.

SparkQA · 2020-07-08T07:05:03Z

Test build #125295 has finished for PR 28803 at commit 0270bb9.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

sarutak · 2020-07-13T20:05:36Z

I am +0 with the proposal. The pagination makes the visualization simpler, but also makes the UI usage more complex. If you insist, let's merge this one.

I wondered the pagination feature for task timeline is one compromise so I came up with bringing the idea into jobs timeline but this solution has both upside and downside as you suggested.
So, do you have any better idea to avoid being the timeline noisy?

github-actions · 2020-10-22T00:55:44Z

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!

Add pagination support for all jobs timeline.

90b132b

probot-autolabeler bot added CORE WEB UI labels Jun 11, 2020

sarutak added 2 commits June 13, 2020 14:28

Merge branch 'master' of https://github.com/apache/spark into add-pag…

1571d9a

…ination-support-for-all-jobs-timeline

Fix for the case of eventTimelineJobPageSize <= 0.

0270bb9

gengliangwang reviewed Jul 8, 2020

View reviewed changes

github-actions bot added the Stale label Oct 22, 2020

github-actions bot closed this Oct 23, 2020

[SPARK-31971][WEBUI] Add pagination support for all jobs timeline #28803

[SPARK-31971][WEBUI] Add pagination support for all jobs timeline #28803

Uh oh!

Conversation

sarutak commented Jun 11, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

sarutak commented Jun 11, 2020

Uh oh!

gengliangwang commented Jun 11, 2020

Uh oh!

gengliangwang commented Jun 11, 2020

Uh oh!

SparkQA commented Jun 11, 2020

Uh oh!

gengliangwang commented Jun 12, 2020

Uh oh!

sarutak commented Jun 12, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gengliangwang commented Jun 12, 2020

Uh oh!

gengliangwang commented Jun 12, 2020

Uh oh!

sarutak commented Jun 12, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gengliangwang commented Jun 12, 2020

Uh oh!

gengliangwang commented Jun 12, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sarutak commented Jun 12, 2020

Uh oh!

sarutak commented Jun 13, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SparkQA commented Jun 13, 2020

Uh oh!

sarutak commented Jun 13, 2020

Uh oh!

SparkQA commented Jun 13, 2020

Uh oh!

sarutak commented Jul 1, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sarutak commented Jul 8, 2020

Uh oh!

gengliangwang commented Jul 8, 2020

Uh oh!

gengliangwang Jul 8, 2020

Choose a reason for hiding this comment

Uh oh!

sarutak Jul 8, 2020

Choose a reason for hiding this comment

Uh oh!

gengliangwang Jul 8, 2020

Choose a reason for hiding this comment

Uh oh!

sarutak Jul 8, 2020

Choose a reason for hiding this comment

Uh oh!

gengliangwang commented Jul 8, 2020

Uh oh!

SparkQA commented Jul 8, 2020

Uh oh!

sarutak commented Jul 13, 2020

Uh oh!

github-actions bot commented Oct 22, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

sarutak commented Jun 11, 2020 •

edited

Loading

sarutak commented Jun 12, 2020 •

edited

Loading

sarutak commented Jun 12, 2020 •

edited

Loading

gengliangwang commented Jun 12, 2020 •

edited

Loading

sarutak commented Jun 13, 2020 •

edited

Loading

sarutak commented Jul 1, 2020 •

edited

Loading