Skip to content

Commit 1dd4792

Browse files
shahidki31Jackey Lee
authored andcommitted
[SPARK-25566][SPARK-25567][WEBUI][SQL] Support pagination for SQL tab to avoid OOM
## What changes were proposed in this pull request? Currently SQL tab in the WEBUI doesn't support pagination. Because of that following issues are happening. 1) For large number of executions, SQL page is throwing OOM exception (around 40,000) 2) For large number of executions, loading SQL page is taking time. 3) Difficult to analyse the execution table for large number of execution. [Note: spark.sql.ui.retainedExecutions = 50000] All the tabs, Jobs, Stages etc. supports pagination. So, to make it consistent with other tabs SQL tab also should support pagination. I have followed the similar flow of the pagination code in the Jobs and Stages page for SQL page. Also, this patch doesn't make any behavior change for the SQL tab except the pagination support. ## How was this patch tested? bin/spark-shell --conf spark.sql.ui.retainedExecutions=50000 Run 50,000 sql queries. **Before this PR** ![screenshot from 2018-10-05 23-48-27](https://user-images.githubusercontent.com/23054875/46552750-4ed82480-c8f9-11e8-8b05-d60bedddd1b8.png) ![screenshot from 2018-10-05 22-58-11](https://user-images.githubusercontent.com/23054875/46550276-33b5e680-c8f2-11e8-9e32-9ae9c5b181e0.png) **After this PR** Loading of the page is faster, and OOM issue doesn't happen. ![screenshot from 2018-10-05 23-50-32](https://user-images.githubusercontent.com/23054875/46552814-8050f000-c8f9-11e8-96e9-42502d2cfaea.png) Closes apache#22645 from shahidki31/SPARK-25566. Authored-by: Shahid <[email protected]> Signed-off-by: Sean Owen <[email protected]>
1 parent 2de86bb commit 1dd4792

File tree

2 files changed

+312
-96
lines changed

2 files changed

+312
-96
lines changed

core/src/main/scala/org/apache/spark/ui/PagedTable.scala

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,7 @@ import org.apache.spark.util.Utils
3131
*
3232
* @param pageSize the number of rows in a page
3333
*/
34-
private[ui] abstract class PagedDataSource[T](val pageSize: Int) {
34+
private[spark] abstract class PagedDataSource[T](val pageSize: Int) {
3535

3636
if (pageSize <= 0) {
3737
throw new IllegalArgumentException("Page size must be positive")
@@ -72,7 +72,7 @@ private[ui] case class PageData[T](totalPage: Int, data: Seq[T])
7272
/**
7373
* A paged table that will generate a HTML table for a specified page and also the page navigation.
7474
*/
75-
private[ui] trait PagedTable[T] {
75+
private[spark] trait PagedTable[T] {
7676

7777
def tableId: String
7878

0 commit comments

Comments
 (0)