[SPARK-25566][SPARK-25567][WEBUI][SQL]Support pagination for SQL tab to avoid OOM #22645

shahidki31 · 2018-10-05T17:45:11Z

What changes were proposed in this pull request?

Currently SQL tab in the WEBUI doesn't support pagination. Because of that following issues are happening.

For large number of executions, SQL page is throwing OOM exception (around 40,000)
For large number of executions, loading SQL page is taking time.
Difficult to analyse the execution table for large number of execution.
[Note: spark.sql.ui.retainedExecutions = 50000]

All the tabs, Jobs, Stages etc. supports pagination. So, to make it consistent with other tabs
SQL tab also should support pagination.

I have followed the similar flow of the pagination code in the Jobs and Stages page for SQL page.
Also, this patch doesn't make any behavior change for the SQL tab except the pagination support.

How was this patch tested?

bin/spark-shell --conf spark.sql.ui.retainedExecutions=50000
Run 50,000 sql queries.
Before this PR

After this PR

Loading of the page is faster, and OOM issue doesn't happen.

[Spark Job History] SQL UI Page does not support Pagination

shahidki31 · 2018-10-05T18:22:43Z

cc @vanzin @srowen @cloud-fan @dongjoon-hyun . Kindly review the PR.

shahidki31 · 2018-10-06T04:17:44Z

Test step to reproduce OOM without the PR.

bin/spark-shell --conf spark.sql.ui.retainedExecutions=50000
for (i <- 0 until 50000) {
val df = Seq(
(1, 1),
(2, 2)
).toDF()
df.collect()

Without the PR:

After fix:

core/src/main/scala/org/apache/spark/ui/PagedTable.scala

… with intercept with L1 regularization 1 min 10 sec

srowen · 2018-10-06T17:06:35Z

sql/core/src/main/scala/org/apache/spark/sql/execution/ui/AllExecutionsPage.scala

 import org.apache.spark.JobExecutionStatus
 import org.apache.spark.internal.Logging
-import org.apache.spark.ui.{UIUtils, WebUIPage}
+import org.apache.spark.ui._


Nit: can we avoid wildcard imports? we usually avoid them unless there are a very large number of classes to import

Yes. done. Thanks

srowen · 2018-10-06T17:08:04Z

sql/core/src/main/scala/org/apache/spark/sql/execution/ui/AllExecutionsPage.scala

+    val executionPrevPageSize = Option(parameterExecutionPrevPageSize).map(_.toInt).
+      getOrElse(executionPageSize)
+
+    val page: Int = {


Trivial nit: you don't need the type here or an extra block

Done. Thanks

srowen · 2018-10-06T17:14:04Z

sql/core/src/main/scala/org/apache/spark/sql/execution/ui/AllExecutionsPage.scala

Can this be private? and how about using interpolation for the whole thing?

Done. Thanks

srowen · 2018-10-06T17:14:22Z

sql/core/src/main/scala/org/apache/spark/sql/execution/ui/AllExecutionsPage.scala

Here and below, how about interpolation consistently?

Done. Thanks

srowen · 2018-10-06T17:15:13Z

sql/core/src/main/scala/org/apache/spark/sql/execution/ui/AllExecutionsPage.scala

This might be worth using case expressions to clarify what _3 and _1 are.

Thanks. Done.

sql/core/src/main/scala/org/apache/spark/sql/execution/ui/AllExecutionsPage.scala

srowen · 2018-10-06T17:16:57Z

sql/core/src/main/scala/org/apache/spark/sql/execution/ui/AllExecutionsPage.scala

Nit: pull this onto previous line? the indent is odd, at least

Done. Thanks.

srowen · 2018-10-06T17:17:06Z

sql/core/src/main/scala/org/apache/spark/sql/execution/ui/AllExecutionsPage.scala

I believe this shouldn't be indented more

Yes. I have used a different scalastyle and checkstyle xml in IJ it seems.

srowen · 2018-10-06T17:18:12Z

sql/core/src/main/scala/org/apache/spark/sql/execution/ui/AllExecutionsPage.scala

Same, here and around here do you really want to filter first instead of flatMap?

Thanks. I have modified using filter

shahidki31 · 2018-10-06T20:03:07Z

Thank you @srowen for the review. I have addressed the comments.

srowen · 2018-10-06T20:29:17Z

sql/core/src/main/scala/org/apache/spark/sql/execution/ui/AllExecutionsPage.scala

This might be a good place to wrap these to line up the elements in the Seq, one one each line (?) to make it easier to read its contents. They all seem to have an empty string as the second element? can it be removed or did I overlook something? Also you don't need the type here.

Yes. I have modified based on your suggestions. Thanks.

srowen · 2018-10-06T20:29:26Z

sql/core/src/main/scala/org/apache/spark/sql/execution/ui/AllExecutionsPage.scala

Nit: pull this onto previous line

Done. Thanks

srowen · 2018-10-06T20:29:37Z

sql/core/src/main/scala/org/apache/spark/sql/execution/ui/AllExecutionsPage.scala

Nit: space after if. Really, maybe just use require

Done. used require. Thanks

srowen · 2018-10-06T20:30:24Z

sql/core/src/main/scala/org/apache/spark/sql/execution/ui/AllExecutionsPage.scala

.map { case (title, _, _) => title }?

Updated. Thanks.

sql/core/src/main/scala/org/apache/spark/sql/execution/ui/AllExecutionsPage.scala

shahidki31 · 2018-10-06T22:05:51Z

Thank you @srowen , I have modified the code based on your suggestions.

sql/core/src/main/scala/org/apache/spark/sql/execution/ui/AllExecutionsPage.scala

shahidki31 · 2018-10-08T18:26:44Z

Hi @srowen , There is one behavior change this PR introduces, which is correct. Sorting Job Ids in the previous versions of spark was not proper. After the PR the sorting is proper.

Previous versions of spark:

After the PR:

I have updated the code accordingly.

felixcheung · 2018-10-09T06:12:37Z

ok to test

SparkQA · 2018-10-09T06:14:52Z

Test build #97144 has finished for PR 22645 at commit e6af675.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

felixcheung · 2018-10-09T06:20:59Z

@shahidki31 ^^

shahidki31 · 2018-10-09T06:46:16Z

Hi @felixcheung , I will update the code

shahidki31 · 2018-10-09T07:33:05Z

@felixcheung I build locally, Now scalastyle issue is not happening. kindly re-trigger the PR builder.

SparkQA · 2018-10-09T10:50:02Z

Test build #97146 has finished for PR 22645 at commit 2b5a724.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

shahidki31 · 2018-10-09T11:35:59Z

@felixcheung It is a random failure. Could you please re-trigger the test. Thanks
Please refer:
https://issues.apache.org/jira/browse/SPARK-23622

SparkQA · 2018-10-09T18:39:57Z

Test build #4361 has finished for PR 22645 at commit 2b5a724.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

srowen · 2018-10-12T17:37:50Z

Merged to master

shahidki31 · 2018-10-12T17:55:37Z

Thanks a lot @srowen

cloud-fan · 2018-10-13T05:53:29Z

I found the UI patches are very hard to review, because we embed HTML/Javascript in Scala code. Is there a plan to rewrite the Spark UI with some modern frontend frameworks?

shahidki31 · 2018-10-13T06:49:21Z

Hi @cloud-fan , Since other webui pages like jobs, stages, tasks etc. embed the javascript code in scala code, that is why I followed the same. It would be great if we rewrite the spark UI with some modern front end frameworks.

Also, after this PR, loading time of the SQL page improved significantly. earlier loading 55,000 queries sql page took ~4.34 mins from the history server, but after the PR took only ~4 sec.

… to avoid OOM ## What changes were proposed in this pull request? Currently SQL tab in the WEBUI doesn't support pagination. Because of that following issues are happening. 1) For large number of executions, SQL page is throwing OOM exception (around 40,000) 2) For large number of executions, loading SQL page is taking time. 3) Difficult to analyse the execution table for large number of execution. [Note: spark.sql.ui.retainedExecutions = 50000] All the tabs, Jobs, Stages etc. supports pagination. So, to make it consistent with other tabs SQL tab also should support pagination. I have followed the similar flow of the pagination code in the Jobs and Stages page for SQL page. Also, this patch doesn't make any behavior change for the SQL tab except the pagination support. ## How was this patch tested? bin/spark-shell --conf spark.sql.ui.retainedExecutions=50000 Run 50,000 sql queries. **Before this PR** ![screenshot from 2018-10-05 23-48-27](https://user-images.githubusercontent.com/23054875/46552750-4ed82480-c8f9-11e8-8b05-d60bedddd1b8.png) ![screenshot from 2018-10-05 22-58-11](https://user-images.githubusercontent.com/23054875/46550276-33b5e680-c8f2-11e8-9e32-9ae9c5b181e0.png) **After this PR** Loading of the page is faster, and OOM issue doesn't happen. ![screenshot from 2018-10-05 23-50-32](https://user-images.githubusercontent.com/23054875/46552814-8050f000-c8f9-11e8-96e9-42502d2cfaea.png) Closes apache#22645 from shahidki31/SPARK-25566. Authored-by: Shahid <[email protected]> Signed-off-by: Sean Owen <[email protected]>

### What changes were proposed in this pull request? Supports pagination for SQL Statisitcs table in the JDBC/ODBC tab using existing Spark pagination framework. ### Why are the changes needed? It will easier for user to analyse the table and it may fix the potential issues like oom while loading the page, that may occur similar to the SQL page (refer #22645) ### Does this PR introduce any user-facing change? There will be no change in the `SQLStatistics` table in JDBC/ODBC server page execpt pagination support. ### How was this patch tested? Manually verified. Before PR: ![Screenshot 2019-10-22 at 11 37 29 PM](https://user-images.githubusercontent.com/23054875/67316080-73636680-f525-11e9-91bc-ff7e06e3736d.png) After PR: ![Screenshot 2019-10-22 at 10 33 00 PM](https://user-images.githubusercontent.com/23054875/67316092-778f8400-f525-11e9-93f8-1e2815abd66f.png) Closes #26215 from shahidki31/jdbcPagination. Authored-by: shahid <[email protected]> Signed-off-by: Sean Owen <[email protected]>

… JDBC/ODBC Session page ### What changes were proposed in this pull request? In the PR #26215, we supported pagination for sqlstats table in JDBC/ODBC server page. In this PR, we are extending the support of pagination to sqlstats session table by making use of existing pagination classes in #26215. ### Why are the changes needed? Support pagination for sqlsessionstats table in JDBC/ODBC server page in the WEBUI. It will easier for user to analyse the table and it may fix the potential issues like oom while loading the page, that may occur similar to the SQL page (refer #22645) ### Does this PR introduce any user-facing change? There will be no change in the sqlsessionstats table in JDBC/ODBC server page execpt pagination support. ### How was this patch tested? Manually verified. Before: ![Screenshot 2019-10-24 at 11 32 27 PM](https://user-images.githubusercontent.com/23054875/67512507-96715000-f6b6-11e9-9f1f-ab1877eb24e6.png) After: ![Screenshot 2019-10-24 at 10 58 53 PM](https://user-images.githubusercontent.com/23054875/67512314-295dba80-f6b6-11e9-9e3e-dd50c6e62fe9.png) Closes #26246 from shahidki31/SPARK_29589. Authored-by: shahid <[email protected]> Signed-off-by: Sean Owen <[email protected]>

… JDBC/ODBC Session page In the PR apache#26215, we supported pagination for sqlstats table in JDBC/ODBC server page. In this PR, we are extending the support of pagination to sqlstats session table by making use of existing pagination classes in apache#26215. Support pagination for sqlsessionstats table in JDBC/ODBC server page in the WEBUI. It will easier for user to analyse the table and it may fix the potential issues like oom while loading the page, that may occur similar to the SQL page (refer apache#22645) There will be no change in the sqlsessionstats table in JDBC/ODBC server page execpt pagination support. Manually verified. Before: ![Screenshot 2019-10-24 at 11 32 27 PM](https://user-images.githubusercontent.com/23054875/67512507-96715000-f6b6-11e9-9f1f-ab1877eb24e6.png) After: ![Screenshot 2019-10-24 at 10 58 53 PM](https://user-images.githubusercontent.com/23054875/67512314-295dba80-f6b6-11e9-9e3e-dd50c6e62fe9.png) Closes apache#26246 from shahidki31/SPARK_29589. Authored-by: shahid <[email protected]> Signed-off-by: Sean Owen <[email protected]> Signed-off-by: dengziming <[email protected]>

…pache.spark.ui  ### _Why are the changes needed?_  In order to compaible with both Spark-2 and Spark-3, we should respect package `ui`, since apache/spark#22645, some class cope changed from package `ui` to `spark`. ### _How was this patch tested?_ Pass CI Closes #1430 from ulysses-you/spark-2.4. Closes #1429 8b40c60 [ulysses-you] nit f875cda [ulysses-you] ui Authored-by: ulysses-you <[email protected]> Signed-off-by: ulysses-you <[email protected]>

SPARK-25566

4dc1b22

[Spark Job History] SQL UI Page does not support Pagination

shahidki31 force-pushed the SPARK-25566 branch from e2b45d5 to a144270 Compare October 5, 2018 17:49

shahidki31 mentioned this pull request Oct 6, 2018

[SPARK-25575][WEBUI][FOLLOWUP]SQL tab in the spark UI support hide tables #22650

Closed

shahidki31 force-pushed the SPARK-25566 branch 3 times, most recently from 4cce23b to c48077c Compare October 6, 2018 05:03

felixcheung reviewed Oct 6, 2018

View reviewed changes

core/src/main/scala/org/apache/spark/ui/PagedTable.scala Outdated Show resolved Hide resolved

[SPARK-25566]SQL UI Page support Pagination to avoid OOM

4790b37

shahidki31 force-pushed the SPARK-25566 branch from c48077c to 4790b37 Compare October 6, 2018 07:42

[SPARK-25623]LogisticRegressionSuite: multinomial logistic regression…

ff16b07

… with intercept with L1 regularization 1 min 10 sec

srowen requested changes Oct 6, 2018

View reviewed changes

shahidki31 force-pushed the SPARK-25566 branch 9 times, most recently from cf3a186 to 98e46ea Compare October 6, 2018 20:00

srowen requested changes Oct 6, 2018

View reviewed changes

shahidki31 force-pushed the SPARK-25566 branch 3 times, most recently from d9df4ba to 2a3c534 Compare October 6, 2018 22:00

shahidki31 force-pushed the SPARK-25566 branch from 2a3c534 to 6993b1a Compare October 6, 2018 22:12

felixcheung reviewed Oct 7, 2018

View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/execution/ui/AllExecutionsPage.scala Outdated Show resolved Hide resolved

sort job Id

e6af675

scalastyle

2b5a724

asfgit closed this in 3494b12 Oct 12, 2018

shahidki31 mentioned this pull request Oct 22, 2019

[SPARK-29559][Webui] Support pagination for JDBC/ODBC Server page #26215

Closed

shahidki31 mentioned this pull request Oct 24, 2019

[SPARK-29589][WEBUI] Support pagination for sqlstats session table in JDBC/ODBC Session page #26246

Closed

ulysses-you mentioned this pull request Nov 22, 2021

[KYUUBI #1429][TASK-1] Move ui package from org.apache.spark to org.apache.spark.ui apache/kyuubi#1430

Closed

[SPARK-25566][SPARK-25567][WEBUI][SQL]Support pagination for SQL tab to avoid OOM #22645

[SPARK-25566][SPARK-25567][WEBUI][SQL]Support pagination for SQL tab to avoid OOM #22645

Uh oh!

Conversation

shahidki31 commented Oct 5, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

shahidki31 commented Oct 5, 2018

Uh oh!

shahidki31 commented Oct 6, 2018

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shahidki31 commented Oct 6, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

shahidki31 commented Oct 6, 2018

Uh oh!

Uh oh!

shahidki31 commented Oct 8, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

felixcheung commented Oct 9, 2018

Uh oh!

SparkQA commented Oct 9, 2018

Uh oh!

felixcheung commented Oct 9, 2018

Uh oh!

shahidki31 commented Oct 9, 2018

shahidki31 commented Oct 5, 2018 •

edited

Loading

shahidki31 commented Oct 8, 2018 •

edited

Loading

shahidki31 commented Oct 13, 2018 •

edited

Loading