[SPARK-33287][SS][UI]Expose state custom metrics information on SS UI #30336

gaborgsomogyi · 2020-11-11T12:53:28Z

What changes were proposed in this pull request?

Structured Streaming UI is not containing state custom metrics information. In this PR I've added it.

Why are the changes needed?

Missing state custom metrics information.

Does this PR introduce any user-facing change?

Additional UI elements appear.

How was this patch tested?

Existing unit tests + manual test.

#Compile Spark
echo "spark.sql.streaming.ui.enabledCustomMetricList stateOnCurrentVersionSizeBytes" >> conf/spark-defaults.conf
sbin/start-master.sh
sbin/start-worker.sh spark://gsomogyi-MBP16:7077
./bin/spark-submit --master spark://gsomogyi-MBP16:7077 --deploy-mode client --class com.spark.Main ../spark-test/target/spark-test-1.0-SNAPSHOT-jar-with-dependencies.jar

SparkQA · 2020-11-11T14:17:49Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35544/

SparkQA · 2020-11-11T14:38:48Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35544/

HyukjinKwon · 2020-11-11T15:14:57Z

cc @gengliangwang, @HeartSaVioR and @xuanyuanking FYI

gaborgsomogyi · 2020-11-11T15:28:48Z

Thanks pinging peoples, just wanted to wait until tests pass.

SparkQA · 2020-11-11T17:45:58Z

Test build #130939 has finished for PR 30336 at commit 2eefeb2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

sql/core/src/main/scala/org/apache/spark/sql/streaming/ui/StreamingQueryStatisticsPage.scala

sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala

xuanyuanking

Thanks for the work, will also test offline! cc @uncleGen @zsxwing

sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala

SparkQA · 2020-11-13T14:01:39Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35668/

SparkQA · 2020-11-13T14:28:18Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35668/

SparkQA · 2020-11-13T16:18:09Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35676/

SparkQA · 2020-11-13T16:48:01Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35676/

SparkQA · 2020-11-13T17:47:44Z

Test build #131074 has finished for PR 30336 at commit 19e2da1.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-11-13T18:13:09Z

Test build #131072 has finished for PR 30336 at commit f11bc22.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-11-13T18:23:39Z

Test build #131064 has finished for PR 30336 at commit 6654934.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HeartSaVioR

One comment on the location of config, otherwise looks OK. Manually verified the functionality as well.

core/src/main/scala/org/apache/spark/internal/config/UI.scala

HyukjinKwon · 2020-11-16T01:44:09Z

Oh, @sarutak too FYI

sql/core/src/main/scala/org/apache/spark/sql/streaming/ui/StreamingQueryStatisticsPage.scala

core/src/main/scala/org/apache/spark/internal/config/UI.scala

sql/core/src/main/scala/org/apache/spark/sql/streaming/ui/StreamingQueryStatisticsPage.scala

sarutak · 2020-11-16T09:40:18Z

Hi @gaborgsomogyi , you'd like to add this new feature but you say you tested this feature manually so could you explain how you tested to us in the description? It helps reviewers.

gaborgsomogyi · 2020-11-16T10:32:49Z

@sarutak added how I've tested.

sarutak · 2020-11-16T11:35:47Z

@gaborgsomogyi I expected what the streaming query is (code snippet may be enough) and what you confirmed by the manual test... With those information, we can understand what you did (and did not) test.
Here is quoted from the PR template.

If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.

sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStore.scala

SparkQA · 2020-11-19T14:03:24Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35953/

SparkQA · 2020-11-19T14:29:50Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35953/

SparkQA · 2020-11-19T18:51:49Z

Test build #131349 has finished for PR 30336 at commit d43b1b9.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class StateStoreCustomSumMetric(name: String, desc: String, override val unit: String)
case class StateStoreCustomSizeMetric(name: String, desc: String, override val unit: String)
case class StateStoreCustomTimingMetric(name: String, desc: String, override val unit: String)

sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStore.scala

SparkQA · 2020-11-20T10:26:57Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36018/

SparkQA · 2020-11-20T10:50:12Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36018/

SparkQA · 2020-11-20T13:15:00Z

Test build #131412 has finished for PR 30336 at commit 9a87255.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class StateStoreCustomSumMetric(name: String, desc: String) extends StateStoreCustomMetric
case class StateStoreCustomSizeMetric(name: String, desc: String) extends StateStoreCustomMetric
case class StateStoreCustomTimingMetric(name: String, desc: String) extends StateStoreCustomMetric

HeartSaVioR · 2020-11-21T01:57:28Z

@gaborgsomogyi Could you please fix the merge conflict?

@xuanyuanking
This PR reverted the API change which would address your review comment. Could you please verify on your own? It's appreciated if you have more comments, but I see this was the only concern.
#30336 (comment)

Once merge conflict is fixed, I'll merge in early next week unless there's no further comment.

SparkQA · 2020-11-21T12:37:37Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36069/

SparkQA · 2020-11-21T13:01:57Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36069/

SparkQA · 2020-11-21T16:18:21Z

Test build #131463 has finished for PR 30336 at commit 864062f.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class GetShufflePushMergerLocations(numMergersNeeded: Int, hostsToFilter: Set[String])
case class RemoveShufflePushMergerLocation(host: String) extends ToBlockManagerMaster
case class ParseUrl(children: Seq[Expression], failOnError: Boolean = SQLConf.get.ansiEnabled)

sql/core/src/main/scala/org/apache/spark/sql/streaming/ui/StreamingQueryStatisticsPage.scala

xuanyuanking · 2020-11-22T00:51:16Z

LGTM, @HeartSaVioR thanks for pinging me!

SparkQA · 2020-11-24T15:48:32Z

Test build #131657 has finished for PR 30336 at commit 115de28.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

gaborgsomogyi · 2020-11-24T15:53:28Z

retest this please

SparkQA · 2020-11-24T16:43:54Z

Test build #131676 has finished for PR 30336 at commit 115de28.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-11-24T18:11:22Z

Test build #131690 has started for PR 30336 at commit e2f1594.

HeartSaVioR · 2020-11-24T22:37:54Z

I'll merge this as Github Action passed for the last change (e2f1594). Jenkins seems to have been unstable and I don't like to drag it anymore due to this.

HeartSaVioR · 2020-11-24T22:39:41Z

Thanks all for reviewing and thanks @gaborgsomogyi for the contribution. Merged to master.

gaborgsomogyi · 2020-11-25T06:12:41Z

Thank you all for taking care!

[SPARK-33287][SS][UI]Expose state custom metrics information on SS UI

2eefeb2

github-actions bot added SQL STRUCTURED STREAMING WEB UI labels Nov 11, 2020

HeartSaVioR reviewed Nov 12, 2020

View reviewed changes

xuanyuanking reviewed Nov 12, 2020

View reviewed changes

sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala Outdated Show resolved Hide resolved

Partial review fixes

6654934

gaborgsomogyi force-pushed the SPARK-33287 branch from 082ec87 to 6654934 Compare November 13, 2020 13:06

Switch to enable flag

f11bc22

github-actions bot added the CORE label Nov 13, 2020

Cleanup

19e2da1

HeartSaVioR reviewed Nov 15, 2020

View reviewed changes

core/src/main/scala/org/apache/spark/internal/config/UI.scala Outdated Show resolved Hide resolved

xuanyuanking reviewed Nov 16, 2020

View reviewed changes

sarutak reviewed Nov 16, 2020

View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/streaming/ui/StreamingQueryStatisticsPage.scala Show resolved Hide resolved

xuanyuanking reviewed Nov 19, 2020

View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStore.scala Outdated Show resolved Hide resolved

Unit default value

d43b1b9

HeartSaVioR reviewed Nov 19, 2020

View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStore.scala Outdated Show resolved Hide resolved

gaborgsomogyi added 2 commits November 20, 2020 09:38

Merge branch 'master' into SPARK-33287

bc83741

Remove unit + rename a variable

9a87255

Merge branch 'master' into SPARK-33287

864062f

viirya reviewed Nov 21, 2020

View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/streaming/ui/StreamingQueryStatisticsPage.scala Outdated Show resolved Hide resolved

viirya reviewed Nov 21, 2020

View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/streaming/ui/StreamingQueryStatisticsPage.scala Show resolved Hide resolved

Simplify enabled custom metrics check

115de28

viirya approved these changes Nov 24, 2020

View reviewed changes

Ignore case compare

e2f1594

HeartSaVioR closed this in 95b6dab Nov 24, 2020

[SPARK-33287][SS][UI]Expose state custom metrics information on SS UI #30336

[SPARK-33287][SS][UI]Expose state custom metrics information on SS UI #30336

Uh oh!

Conversation

gaborgsomogyi commented Nov 11, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

SparkQA commented Nov 11, 2020

Uh oh!

SparkQA commented Nov 11, 2020

Uh oh!

HyukjinKwon commented Nov 11, 2020

Uh oh!

gaborgsomogyi commented Nov 11, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SparkQA commented Nov 11, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

xuanyuanking left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

SparkQA commented Nov 13, 2020

Uh oh!

SparkQA commented Nov 13, 2020

Uh oh!

SparkQA commented Nov 13, 2020

Uh oh!

SparkQA commented Nov 13, 2020

Uh oh!

SparkQA commented Nov 13, 2020

Uh oh!

SparkQA commented Nov 13, 2020

Uh oh!

SparkQA commented Nov 13, 2020

Uh oh!

HeartSaVioR left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

HyukjinKwon commented Nov 16, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sarutak commented Nov 16, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gaborgsomogyi commented Nov 16, 2020

Uh oh!

sarutak commented Nov 16, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

SparkQA commented Nov 19, 2020

Uh oh!

SparkQA commented Nov 19, 2020

Uh oh!

SparkQA commented Nov 19, 2020

Uh oh!

Uh oh!

SparkQA commented Nov 20, 2020

Uh oh!

SparkQA commented Nov 20, 2020

Uh oh!

SparkQA commented Nov 20, 2020

Uh oh!

HeartSaVioR commented Nov 21, 2020

Uh oh!

SparkQA commented Nov 21, 2020

gaborgsomogyi commented Nov 11, 2020 •

edited

Loading

gaborgsomogyi commented Nov 11, 2020 •

edited

Loading

sarutak commented Nov 16, 2020 •

edited

Loading

sarutak commented Nov 16, 2020 •

edited

Loading