[SPARK-46886][CORE] Enable spark.ui.prometheus.enabled by default
#44912
Conversation
|
Could you review this when you have some time, @viirya ? |
| Executor-level metrics are sent from each executor to the driver as part of the Heartbeat to describe the performance metrics of Executor itself like JVM heap memory, GC information. | ||
| Executor metric values and their measured memory peak values per executor are exposed via the REST API in JSON format and in Prometheus format. | ||
| The JSON end point is exposed at: `/applications/[app-id]/executors`, and the Prometheus endpoint at: `/metrics/executors/prometheus`. | ||
| The Prometheus endpoint is conditional to a configuration parameter: `spark.ui.prometheus.enabled=true` (the default is `false`). |
There was a problem hiding this comment.
Hmm, why we need to remove it from document?
There was a problem hiding this comment.
The previous sentence already mentioned this endpoint. And, this sentence is invalid because we enabled it by default, @viirya .
There was a problem hiding this comment.
Oh, I means that the config now is not mentioned in this doc. Is any other place we have mentioned this config?
There was a problem hiding this comment.
Ah, for that, I can add it to the config table officially.
There was a problem hiding this comment.
Actually, a few other UI configs are missing too. I can handle them together.
|
Thank you, @viirya ! For the UI config PR, I'll make a PR today. |
|
Merged to master for Apache Spark 4.0.0. |
|
Thank you @dongjoon-hyun . Please pine me once you create the doc PR. |
|
Sure! I'm make a PR in 5 minutes. |
### What changes were proposed in this pull request? `spark.ui.prometheus.enabled` has been used since Apache Spark 3.0.0. - apache#25770 This PR aims to enable `spark.ui.prometheus.enabled` by default like Driver `JSON` API in Apache Spark 4.0.0. | | JSON End Point | Prometheus End Point | | ------- | ------------------------------------ | --------------------------------- | | Driver | /api/v1/applications/{id}/executors/ | /metrics/executors/prometheus/ | ### Why are the changes needed? **BEFORE** ``` $ bin/spark-shell $ curl -s http://localhost:4040/metrics/executors/prometheus | wc -l 0 ``` **AFTER** ``` $ bin/spark-shell $ curl -s http://localhost:4040/metrics/executors/prometheus | wc -l 20 ``` ### Does this PR introduce _any_ user-facing change? No, this is only a new endpoint. ### How was this patch tested? Pass the CIs and do manual test. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#44912 from dongjoon-hyun/SPARK-46886. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
2 participants
What changes were proposed in this pull request?
spark.ui.prometheus.enabledhas been used since Apache Spark 3.0.0.This PR aims to enable
spark.ui.prometheus.enabledby default like DriverJSONAPI in Apache Spark 4.0.0.Why are the changes needed?
BEFORE
AFTER
Does this PR introduce any user-facing change?
No, this is only a new endpoint.
How was this patch tested?
Pass the CIs and do manual test.
Was this patch authored or co-authored using generative AI tooling?
No.