[SPARK-21809] : Change Stage Page to use datatables to support sorting columns and searching #21688

pgandhi999 · 2018-07-01T19:28:58Z

Support column sort, pagination and search for Stage Page using jQuery DataTable and REST API. Before this commit, the Stage page generated a hard-coded HTML table that could not support search. Supporting search and sort (over all applications rather than the 20 entries in the current page) in any case will greatly improve the user experience.
Created the stagespage-template.html for displaying application information in datables. Added REST api endpoint and javascript code to fetch data from the endpoint and display it on the data table.
Because of the above change, certain functionalities in the page had to be modified to support the addition of datatables. For example, the toggle checkbox 'Select All' previously would add the checked fields as columns in the Task table and as rows in the Summary Metrics table, but after the change, only columns are added in the Task Table as it got tricky to add rows dynamically in the datatables.

How was this patch tested?

I have attached the screenshots of the Stage Page UI before and after the fix.
Before:

After:

…rspage.js

…g datatables

…ctionality

gatorsmile · 2018-07-01T19:35:08Z

ok to test

pgandhi999 · 2018-07-01T19:35:11Z

@ajbozarth @vanzin @tgravescs Hello,
After our last discussion in PR #19270, I have made some changes to the datatables, to fetch the data and perform sort, search and pagination on the server side for the tasks table. I request you to review the changes.

Also one known issue with the PR is that currently, the task datatables are displaying all the columns(shuffle read metrics, write metrics, shuffle spill memory and disk etc.) regardless of any data being there or not. As of yet, I could not find a better way to hide those columns and display them only when there is data as the data is coming from the server end so performing client side validation was proving to be tricky. Will work on it later in the course of addressing reviews.

SparkQA · 2018-07-01T19:39:47Z

Test build #92515 has finished for PR 21688 at commit d79eb80.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-07-01T19:49:44Z

Test build #92516 has finished for PR 21688 at commit 6a0622e.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-07-02T00:18:00Z

Test build #92517 has finished for PR 21688 at commit 132fce9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

ajbozarth · 2018-07-02T00:58:27Z

I'll see if I get a chance to look at this this week, but with the holiday it may have to wait a week.

pgandhi999 · 2018-07-02T14:26:26Z

No problem, thank you @ajbozarth

pgandhi999 · 2018-07-16T22:17:57Z

Hi @ajbozarth did you get a chance to review it yet? Thank you.

ajbozarth · 2018-07-16T22:28:44Z

Sorry, I have not. I've been really busy with other projects and haven't been in the spark code for a few months. I'll take a look when/if I have time, but it'd be faster for @vanzin or @tgravescs to review at this point.

SparkQA · 2018-09-11T04:33:14Z

Test build #95902 has finished for PR 21688 at commit 573390d.

This patch fails from timeout after a configured wait of `400m`.
This patch merges cleanly.
This patch adds no public classes.

abellina

@pgandhi999 Thanks for the PR. I know this is going to be very useful, as the tables you are improving are really slow to load and sort currently. This is a first pass, so take a look. I should do another pass in more detail, but let me know what you think about some of these suggestions below.

One general nit: For the Loading modal, I see the whole Stage page dims. Could we instead do this in the specific datatable that is being reloaded?

So instead of .blockUI I think we can select an element, and then .block it. I haven't tried this: http://malsup.com/jquery/block/#element

abellina · 2018-09-10T21:43:20Z

core/src/main/resources/org/apache/spark/ui/static/stagepage.js

+        var stageId = words2[1].substr(3, stageIdLen - 3);
+        var newBaseURI = words.slice(0, ind + 2).join('/');
+        return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId;
+    }


I think this is an else.. it's either proxy or history. Though I am fairly certain this function can (should) be rewritten as a regex. Maybe even one that can be configured in the future.

I agree with you on the part about if else, however, we do need the value of ind to determine the stage id and attempt in order to construct our REST endpoint url so, the check if(ind > 0) is still needed. Regarding the regex part, I did not completely get your point, do you have a specific example or something? Thanks.

abellina · 2018-09-10T21:44:46Z

core/src/main/resources/org/apache/spark/ui/static/stagepage.js

+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+    if(columnKey == "executorRunTime") {


I'd replace with a switch statement.

abellina · 2018-09-10T21:48:27Z

core/src/main/resources/org/apache/spark/ui/static/stagepage.js

+        "<span class='expand-input-rate-arrow arrow-closed' id='arrowtoggle1'></span>" +
+        " Show Additional Metrics" +
+        "</a></div>" +
+        "<div class='container-fluid' id='toggle-metrics' hidden>" +


In the current UI, the "Show Additional Metrics" button will go ahead and toggle in "Summary Metrics":

Scheduler Delay
Task Deserialization Time
Result Serialization Time
Getting Result TIme
Peak Execution Memory

But in my local run of this change, the "Summary Metrics" table is static and not changed by the button.

Yes, that is because, each time, we show or hide a row, we have to construct the entire datatable again. Not sure if that is what we want, but we can discuss further.

abellina · 2018-09-11T16:11:01Z

core/src/main/resources/org/apache/spark/ui/static/stagepage.js

+                $('#active-tasks-table_filter input').unbind();
+                $('#active-tasks-table_filter input').bind('keyup', function(e) {
+                  if(e.keyCode == 13) {
+                    taskTableSelector.search( this.value ).draw();


I wasn't expecting the search to wait for me to press enter. I realize this is to do the sorting server side. Did you find that datatables couldn't sort/filter a large task? Since the first load will have the full payload, I am wondering if we should just do front-end sorting/filtering after that

The tasks datatable has been implemented with server-side sorting, pagination and searching. So the data is pushed directly from the server, thus, implementing a front-end filtering functionality is not possible here. Regarding pressing Enter to get the results, I can change it to happen on keypress. Let me know what you think.

I have implemented the search functionality on any keyup rather than Enter in the PR. Thank you.

@pgandhi999 I meant to do this with a timer, so when you finish typing a 5 second timer for example starts (and gets reset on keyup). Only when the timer is up then it should make the fetch.

abellina · 2018-09-11T16:12:12Z

core/src/main/resources/org/apache/spark/ui/static/stagepage.js

+        $.getJSON(endPoint, function(response, status, jqXHR) {
+
+            // prepare data for task aggregated metrics table
+            indices = Object.keys(response[0].executorSummary);


nit, set var responseBody = response[0], and use responseBody throughout.

abellina · 2018-09-11T16:27:51Z

core/src/main/resources/org/apache/spark/ui/static/stagepage.js

+
+                var task_metrics_table = [];
+                var stageAttemptId = getStageAttemptId();
+                $.getJSON(stageEndPoint(appId) + "/"+stageAttemptId+"/taskSummary?quantiles=0,0.25,0.5,0.75,1.0", function(response1, status, jqXHR) {


a lot of code happens inside of this callback. Could each of the tables be broken down into a function? With the callback here reduced to calls to a set of functions?

So this callback takes care of loading the entire task-summary table on client end, and all the tasks depend on the data received in the callback, but we can use functions instead though. Let me know what way works best.

I think functions would be clearer IMHO, it is more of a style thing. That way you could send to each function only what it needs.

abellina · 2018-09-11T16:32:55Z

core/src/main/scala/org/apache/spark/status/AppStatusStore.scala

    ordered.skip(offset).max(length).asScala.map(_.toApi).toSeq
  }

+  // Filters task list based on search parameter


if the search can happen in UI via datatables, this function should go away.

abellina · 2018-09-11T20:40:22Z

core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala

-                <span class="additional-metric-title"><em>(De)select All</em></span>
-            </li>
-            <li>
-              <span data-toggle="tooltip"


did these tooltips go away?

Sorry about that, added those tooltips again. Thanks.

abellina · 2018-09-11T22:03:52Z

core/src/main/scala/org/apache/spark/status/api/v1/StagesResource.scala

-      val ret = ui.store.stageData(stageId, details = details)
+      var ret = ui.store.stageData(stageId, details = details)
      if (ret.nonEmpty) {
+        for (i <- 0 to (ret.length - 1)) {


Could this be rewritten as a range loop for (r <- ret)

I would also think that things like executorSummary.get.get(execId) could be stored somewhere in the stack, e.g. clean up a bit. It's hard to follow what you are setting.

abellina · 2018-09-11T22:15:56Z

core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala

 }

-private[ui] object ApiHelper {
+private[spark] object ApiHelper {


why was this change needed?

In order to use the function indexName in StagesResource.scala.

SparkQA · 2018-09-14T22:41:50Z

Test build #96081 has finished for PR 21688 at commit af83b6e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-09-17T19:39:45Z

Test build #96146 has finished for PR 21688 at commit 07c1f19.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

tgravescs

still reviewing in detail, tried it out and left some comments from testing.

It would be nice if we could find a way to leave out the columns not needed. Fo rinstance if we aren't outputting anything leave out the "outupt Size/records" columns

tgravescs · 2018-09-18T20:26:16Z

core/src/main/resources/org/apache/spark/ui/static/stagepage.js

+} );
+
+// This function will only parse the URL under certain format
+// e.g. https://domain:50509/history/application_1502220952225_59143/stages/stage/?id=0&attempt=0


might be nice to add example when used with proxy

tgravescs · 2018-09-18T20:27:30Z

core/src/main/resources/org/apache/spark/ui/static/stagepage.js

+// e.g. https://domain:50509/history/application_1502220952225_59143/stages/stage/?id=0&attempt=0
+function stageEndPoint(appId) {
+    var words = document.baseURI.split('/');
+    var words2 = document.baseURI.split('?');


rename words2 to something like queryString

tgravescs · 2018-09-18T20:36:34Z

core/src/main/resources/org/apache/spark/ui/static/stagepage.js

+    var ind = words.indexOf("proxy");
+    if (ind > 0) {
+        var appId = words[ind + 1];
+        var stageIdLen = words2[1].indexOf('&');


seems a bit brittle if the parameters were to change order, could you do a regexp for like id=[0-9]+[^&]*

tgravescs · 2018-09-19T16:33:29Z

core/src/main/resources/org/apache/spark/ui/static/stagepage.js

+                $.getJSON(stageEndPoint(appId) + "/"+stageAttemptId+"/taskSummary?quantiles="+quantiles, function(taskMetricsResponse, status, jqXHR) {
+                    var taskMetricIndices = Object.keys(taskMetricsResponse);
+                    taskMetricIndices.forEach(function (ix) {
+                        var columnName = getColumnNameForTaskMetricSummary(ix);


the summary metrics Metric used to have tooltips on hover

It is somewhat tricky to have tooltips configured for all rows in that particular column, however, we have the same tooltips available in checkboxes and executor summary table columns.

@pgandhi999 I think we mentioned switching on ix in the code below, instead of columnName.

Also could we change ix to be columnKey?

tgravescs · 2018-09-19T16:46:54Z

core/src/main/resources/org/apache/spark/ui/static/stagepage.js

+                            {
+                                data: function (row, type) {
+                                    switch(row.metric) {
+                                        case 'Input Size / Records':


missing tooltip

Discussed in previous comment

tgravescs · 2018-09-19T16:51:53Z

core/src/main/resources/org/apache/spark/ui/static/stagespage-template.html

+                <th>Output:</th>
+                <th>Shuffle Read: </th>
+                <th>Shuffle Write: </th>
+                <th>Shuffle Spill (Memory): </th>


tooltips missing from this and a few of the others

tgravescs · 2018-09-19T16:58:44Z

core/src/main/resources/org/apache/spark/ui/static/stagespage-template.html

+                </thead>
+                <tbody>
+                </tbody>
+            </table>


summary metrics table shows a bit odd when no tasks have finished, might not be a big deal but could see if we can improve. Basically it doesn't show any rows.

When I tested it, it did show me rows, so if you can share an example, that would be nice. Thank you.

tgravescs · 2018-09-19T17:05:19Z

core/src/main/resources/org/apache/spark/ui/static/stagepage.js

+                            { "type": "file-size", "targets": 5 }
+                        ],
+                        "paging": false,
+                        "searching": false,


what is searching for the aggregated metrics table based on? If you search for an integer like 32 (which would match an id) it finds other things but also shows rows with no 32 in it. I also assume the search is based on the unformatted numbers because I tried ot search for 1.2 GB and 1.2 which should have matched on the Input Size but it showed nothing

Yep we can discuss that.

tgravescs · 2018-09-19T17:11:42Z

core/src/main/resources/org/apache/spark/ui/static/stagepage.js

+                var taskTableSelector = $(taskTable).DataTable(task_conf);
+                $('#active-tasks-table_filter input').unbind();
+                $('#active-tasks-table_filter input').bind('keyup', function(e) {
+                  taskTableSelector.search( this.value ).draw();


search on the task table seems to do query upon typing, not sure that is what we want, would be better if use completed and hit enter since its doing server side query

Implemented search on keyup with timeout, thank you.

tgravescs · 2018-09-19T17:13:57Z

core/src/main/resources/org/apache/spark/ui/static/stagepage.js

+                            data : function (row, type) {
+                                if ("taskMetrics" in row) {
+                                    if (row.taskMetrics.shuffleWriteMetrics.writeTime > 0) {
+                                        return type === 'display' ? formatDuration(parseInt(row.taskMetrics.shuffleWriteMetrics.writeTime) / 1000000) : row.taskMetrics.shuffleWriteMetrics.writeTime;


write time shows up with 6 decimal points, we probably don't need that precision here:
36.213747 ms

tgravescs · 2018-09-19T19:17:35Z

core/src/main/resources/org/apache/spark/ui/static/stagepage.js

+$.extend( $.fn.dataTable.ext.type.order, {
+    "file-size-pre": ConvertDurationString,
+
+    "file-size-asc": function ( a, b ) {


file-size name here doesn't match converting duration. if we can change the name to be duration- instead of file-size might make more sense

tgravescs · 2018-09-19T19:18:46Z

core/src/main/resources/org/apache/spark/ui/static/stagepage.js

+                var task_metrics_table = [];
+                var stageAttemptId = getStageAttemptId();
+                var quantiles = "0,0.25,0.5,0.75,1.0";
+                $.getJSON(stageEndPoint(appId) + "/"+stageAttemptId+"/taskSummary?quantiles="+quantiles, function(taskMetricsResponse, status, jqXHR) {


wrap this line at 100 chars if we can

SparkQA · 2018-11-19T22:54:54Z

Test build #99011 has finished for PR 21688 at commit a696dcc.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Changing searchValue from Option to String

tgravescs · 2018-11-20T20:25:57Z

core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala


-  def makeTimeline(tasks: Seq[TaskData], currentTime: Long): Seq[Node] = {
+  def makeTimeline(tasks: Seq[TaskData], currentTime: Long, page: Int, pageSize: Int,
+    totalPages: Int, stageId: Int, stageAttemptId: Int, totalTasks: Int): Seq[Node] = {


fix format of arguments to be multi-line style like:

def runJob( sc: SparkContext, rdd: JavaRDD[Array[Byte]], partitions: JArrayList[Int]): Array[Any] = {

tgravescs · 2018-11-20T20:51:20Z

core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala

    val parameterTaskPage = UIUtils.stripXSS(request.getParameter("task.page"))
    val parameterTaskSortColumn = UIUtils.stripXSS(request.getParameter("task.sort"))
    val parameterTaskSortDesc = UIUtils.stripXSS(request.getParameter("task.desc"))
    val parameterTaskPageSize = UIUtils.stripXSS(request.getParameter("task.pageSize"))


we need to clean this up as its not really used anymore, can you file a jira to change the timeline to use rest api or the data from the other tables..

Done, https://issues.apache.org/jira/browse/SPARK-26130.

SparkQA · 2018-11-21T00:34:23Z

Test build #99076 has finished for PR 21688 at commit b3e2370.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-11-21T01:02:53Z

Test build #99077 has finished for PR 21688 at commit 672b12a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-11-21T04:42:01Z

Test build #99084 has finished for PR 21688 at commit 8f6efbd.

This patch fails from timeout after a configured wait of 400m.
This patch merges cleanly.
This patch adds no public classes.

tgravescs · 2018-11-21T14:23:09Z

test this please

SparkQA · 2018-11-21T17:49:37Z

Test build #99127 has finished for PR 21688 at commit 8f6efbd.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

abellina · 2018-11-21T20:44:05Z

The failing suite is OrcQuerySuite . Looks unrelated.

tgravescs · 2018-11-22T04:59:28Z

Test this please

tgravescs · 2018-11-26T14:20:25Z

Test this please

SparkQA · 2018-11-26T18:55:59Z

Test build #4442 has finished for PR 21688 at commit 8f6efbd.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

tgravescs · 2018-11-26T20:07:41Z

+1 , going to merge to master

There are a few followup jiras on this.

make the timeline visualization better: https://issues.apache.org/jira/browse/SPARK-26130
improve search functionalit: https://issues.apache.org/jira/browse/SPARK-25719

…kList` ## What changes were proposed in this pull request? In the method `taskList`(since #21688), the executor log value is queried in KV store for every task(method `constructTaskData`). This PR propose to use a hashmap for reducing duplicated KV store lookups in the method. ![image](https://user-images.githubusercontent.com/1097932/49946230-841c7680-ff29-11e8-8b83-d8f7553bfe5e.png) ## How was this patch tested? Manual check Closes #23310 from gengliangwang/removeExecutorLog. Authored-by: Gengliang Wang <[email protected]> Signed-off-by: Sean Owen <[email protected]>

…columns and searching Support column sort, pagination and search for Stage Page using jQuery DataTable and REST API. Before this commit, the Stage page generated a hard-coded HTML table that could not support search. Supporting search and sort (over all applications rather than the 20 entries in the current page) in any case will greatly improve the user experience. Created the stagespage-template.html for displaying application information in datables. Added REST api endpoint and javascript code to fetch data from the endpoint and display it on the data table. Because of the above change, certain functionalities in the page had to be modified to support the addition of datatables. For example, the toggle checkbox 'Select All' previously would add the checked fields as columns in the Task table and as rows in the Summary Metrics table, but after the change, only columns are added in the Task Table as it got tricky to add rows dynamically in the datatables. ## How was this patch tested? I have attached the screenshots of the Stage Page UI before and after the fix. **Before:** <img width="1419" alt="30564304-35991e1c-9c8a-11e7-850f-2ac7a347f600" src="https://user-images.githubusercontent.com/22228190/42137915-52054558-7d3a-11e8-8c85-433b2c94161d.png"> <img width="1435" alt="31360592-cbaa2bae-ad14-11e7-941d-95b4c7d14970" src="https://user-images.githubusercontent.com/22228190/42137928-79df500a-7d3a-11e8-9068-5630afe46ff3.png"> **After:** <img width="1432" alt="31360591-c5650ee4-ad14-11e7-9665-5a08d8f21830" src="https://user-images.githubusercontent.com/22228190/42137936-a3fb9f42-7d3a-11e8-8502-22b3897cbf64.png"> <img width="1388" alt="31360604-d266b6b0-ad14-11e7-94b5-dcc4bb5443f4" src="https://user-images.githubusercontent.com/22228190/42137970-0fabc58c-7d3b-11e8-95ad-383b1bd1f106.png"> Closes apache#21688 from pgandhi999/SPARK-21809-2.3. Authored-by: pgandhi <[email protected]> Signed-off-by: Thomas Graves <[email protected]>

…kList` ## What changes were proposed in this pull request? In the method `taskList`(since apache#21688), the executor log value is queried in KV store for every task(method `constructTaskData`). This PR propose to use a hashmap for reducing duplicated KV store lookups in the method. ![image](https://user-images.githubusercontent.com/1097932/49946230-841c7680-ff29-11e8-8b83-d8f7553bfe5e.png) ## How was this patch tested? Manual check Closes apache#23310 from gengliangwang/removeExecutorLog. Authored-by: Gengliang Wang <[email protected]> Signed-off-by: Sean Owen <[email protected]>

### What changes were proposed in this pull request? 1. Remove the fixed width style of class `container-fluid-div`. So that the UI looks clean when the text is long. 2. Add one space between a checkbox and the text on the right side, which is consistent with the stage page. ### Why are the changes needed? The width of class `container-fluid-div` is set as 200px after #21688 . This makes the checkbox in the executor page messy. ![image](https://user-images.githubusercontent.com/1097932/100242069-3bc5ab80-2ee9-11eb-8c7d-96c221398fee.png) We should remove the width limit. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Manual test. After the changes: ![image](https://user-images.githubusercontent.com/1097932/100257802-2f4a4e80-2efb-11eb-9eb0-92d6988ad14b.png) Closes #30500 from gengliangwang/reviseStyle. Authored-by: Gengliang Wang <[email protected]> Signed-off-by: HyukjinKwon <[email protected]>

…eSuite.scala` ### What changes were proposed in this pull request? This pr removes a test file `StagePageSuite.scala` that is no longer needed. ### Why are the changes needed? The current `StagePageSuite.scala` contains one test case and one private method. The reasons for cleaning them up are as follows: 1. The assertion in the test case "ApiHelper.COLUMN_TO_INDEX should match headers of the task table" was cleaned up in SPARK-44490 | #42085. Currently, this test case only creates a `StageData` object and an `AppStatusStore` object, with no other operations or assertions. Moreover, the creation of these objects is also covered in other test cases, such as `KVStoreProtobufSerializerSuite`, `AppStatusStoreSuite`, etc. Therefore, this is already an unnecessary test case and can be cleaned up. 2. The private method `renderStagePage` has not been used since SPARK-21809 | #21688, so it can be cleaned up. After cleaning up the above content, `StagePageSuite.scala` becomes an empty file, so it can be completely deleted. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Pass GitHub Actions ### Was this patch authored or co-authored using generative AI tooling? No Closes #49721 from LuciferYang/SPARK-51026. Authored-by: yangjie01 <[email protected]> Signed-off-by: yangjie01 <[email protected]>

…eSuite.scala` ### What changes were proposed in this pull request? This pr removes a test file `StagePageSuite.scala` that is no longer needed. ### Why are the changes needed? The current `StagePageSuite.scala` contains one test case and one private method. The reasons for cleaning them up are as follows: 1. The assertion in the test case "ApiHelper.COLUMN_TO_INDEX should match headers of the task table" was cleaned up in SPARK-44490 | #42085. Currently, this test case only creates a `StageData` object and an `AppStatusStore` object, with no other operations or assertions. Moreover, the creation of these objects is also covered in other test cases, such as `KVStoreProtobufSerializerSuite`, `AppStatusStoreSuite`, etc. Therefore, this is already an unnecessary test case and can be cleaned up. 2. The private method `renderStagePage` has not been used since SPARK-21809 | #21688, so it can be cleaned up. After cleaning up the above content, `StagePageSuite.scala` becomes an empty file, so it can be completely deleted. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Pass GitHub Actions ### Was this patch authored or co-authored using generative AI tooling? No Closes #49721 from LuciferYang/SPARK-51026. Authored-by: yangjie01 <[email protected]> Signed-off-by: yangjie01 <[email protected]> (cherry picked from commit 3badfd2) Signed-off-by: yangjie01 <[email protected]>

…eSuite.scala` ### What changes were proposed in this pull request? This pr removes a test file `StagePageSuite.scala` that is no longer needed. ### Why are the changes needed? The current `StagePageSuite.scala` contains one test case and one private method. The reasons for cleaning them up are as follows: 1. The assertion in the test case "ApiHelper.COLUMN_TO_INDEX should match headers of the task table" was cleaned up in SPARK-44490 | apache#42085. Currently, this test case only creates a `StageData` object and an `AppStatusStore` object, with no other operations or assertions. Moreover, the creation of these objects is also covered in other test cases, such as `KVStoreProtobufSerializerSuite`, `AppStatusStoreSuite`, etc. Therefore, this is already an unnecessary test case and can be cleaned up. 2. The private method `renderStagePage` has not been used since SPARK-21809 | apache#21688, so it can be cleaned up. After cleaning up the above content, `StagePageSuite.scala` becomes an empty file, so it can be completely deleted. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Pass GitHub Actions ### Was this patch authored or co-authored using generative AI tooling? No Closes apache#49721 from LuciferYang/SPARK-51026. Authored-by: yangjie01 <[email protected]> Signed-off-by: yangjie01 <[email protected]> (cherry picked from commit fb742ff) Signed-off-by: yangjie01 <[email protected]>

pgandhi added 9 commits June 4, 2018 16:00

[SPARK-21809] : Changing file executorspage-template.html and executo…

5450258

…rspage.js

[SPARK-21809] : Making changes in ui resources files

dadc55f

[SPARK-21809] : Getting rquired data in api endpoints

fc88dec

[SPARK-21809] : Rendering the ui compatible with current version usin…

495caaf

…g datatables

Working on Adding Server Side Sort Functionality

c1d41d8

[SPARK-21809] : Implement Server Side Sorting Functionality

567d22b

[SPARK-21809] : Implementing server side pagination and searching fun…

649e95b

…ctionality

[SPARK-21809] : Fixing Tests and task metric table

590e93f

[SPARK-21809] : Refactoring Code and adding shufflereadmetrics data

d79eb80

[SPARK-21809] : Fixing scalastyle tests

6a0622e

[SPARK-21809] : Fixing Scalastyle test again

132fce9

[SPARK-21809] : Changing comment to point to generic url

573390d

abellina suggested changes Sep 11, 2018

View reviewed changes

[SPARK-21809] : Addressing Reviews September 14, 2018

af83b6e

[SPARK-21809] : Doing search on keyup instead of Enter

07c1f19

tgravescs reviewed Sep 19, 2018

View reviewed changes

pgandhi added 2 commits November 20, 2018 14:10

[SPARK-21809] : Adding pagination feature to event timeline graph

b3e2370

[SPARK-21809] : Addressing Reviews November 20, 2018

672b12a

Changing searchValue from Option to String

tgravescs reviewed Nov 20, 2018

View reviewed changes

[SPARK-21809] : Fixing arguments to be multi line style

8f6efbd

asfgit closed this in 76ef02e Nov 26, 2018

srowen mentioned this pull request Dec 3, 2018

[SPARK-26253][WebUI] : Task Summary Metrics Table on Stage Page shows empty table when no data is present #23205

Closed

gengliangwang mentioned this pull request Dec 5, 2018

[SPARK-26281][WebUI] Duration column of task table should be executor run time instead of real duration #23240

Closed

gengliangwang mentioned this pull request Dec 13, 2018

[SPARK-26363][WebUI] Avoid duplicated KV store lookups in method taskList #23310

Closed

TJX2014 mentioned this pull request Jun 25, 2020

[SPARK-32068][WEBUI] Correct task lauchtime show issue due to timezone in stage tab #28918

Closed

gengliangwang mentioned this pull request Nov 25, 2020

[SPARK-33562][UI] Improve the style of the checkbox in executor page #30500

Closed

LuciferYang mentioned this pull request Jan 29, 2025

[SPARK-51026][CORE][TESTS] Remove the unnecessary test file StagePageSuite.scala #49721

Closed

[SPARK-21809] : Change Stage Page to use datatables to support sorting columns and searching #21688

[SPARK-21809] : Change Stage Page to use datatables to support sorting columns and searching #21688

Uh oh!

Conversation

pgandhi999 commented Jul 1, 2018

How was this patch tested?

Uh oh!

gatorsmile commented Jul 1, 2018

Uh oh!

pgandhi999 commented Jul 1, 2018

Uh oh!

SparkQA commented Jul 1, 2018

Uh oh!

SparkQA commented Jul 1, 2018

Uh oh!

SparkQA commented Jul 2, 2018

Uh oh!

ajbozarth commented Jul 2, 2018

Uh oh!

pgandhi999 commented Jul 2, 2018

Uh oh!

pgandhi999 commented Jul 16, 2018

Uh oh!

ajbozarth commented Jul 16, 2018

Uh oh!

SparkQA commented Sep 11, 2018

Uh oh!

abellina left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

abellina left a comment •

edited

Loading