[SPARK-47954][K8S] Support creating ingress entry for external UI access #46184

pan3793 · 2024-04-23T13:02:25Z

What changes were proposed in this pull request?

Ingress a kind of K8s resource to expose HTTP and HTTPS routes from outside the K8s cluster to services within the K8s cluster. It's a common use case that uses ingress to access Spark UI for Spark jobs running on K8s in cluster mode.

This PR aims to add such a built-in feature to allow users to enable Ingress for Spark jobs which running on K8s cluster mode.

State: functionality completed; manuelly tested; ready for review to collect feedback
TODO: UT and docs

Why are the changes needed?

Simplify Spark live UI access from outside of the K8s.

Does this PR introduce any user-facing change?

This PR adds a new feature, but is disabled by default.

How was this patch tested?

I will supply the unit tests later.

Maunnely tested on an internal K8s cluster.

$ build/sbt clean package -Pkubernetes
$ bin/docker-image-tool.sh -r ****** -b java_image_tag=17 -t SPARK-47954 build
$ bin/docker-image-tool.sh -r ****** -t SPARK-47954 push
$ SPARK_PREPEND_CLASSES=true bin/spark-submit \
	--master=k8s://https://***********.org:6443 \
	--deploy-mode cluster \
	--driver-class-path='/opt/spark/examples/jars/*' \
	--driver-memory=512m \
	--executor-memory=512m \
	--executor-cores=1 \
	--num-executors=1 \
	--conf spark.kubernetes.context='*****' \
	--conf spark.kubernetes.authenticate.driver.serviceAccountName=***** \
	--conf spark.kubernetes.namespace=spark \
	--conf spark.kubernetes.file.upload.path=hdfs://*****/spark-staging \
	--conf spark.kubernetes.container.image=******/spark:SPARK-47954 \
	--conf spark.kubernetes.driver.ingress.enabled=true \
	--conf spark.kubernetes.driver.ingress.host={{APP_ID}}.bigdata*********.com \
	--conf spark.kubernetes.driver.ingress.ingressClassName=contour \
	--class org.apache.spark.examples.SparkPi \
	spark-internal 1000

$ kubectl get ingress org-apache-spark-examples-sparkpi-a947e18f0c055185-driver-svc-ingress
NAME                                                                    CLASS     HOSTS                                                         ADDRESS       PORTS   AGE
org-apache-spark-examples-sparkpi-a947e18f0c055185-driver-svc-ingress   contour   spark-cbf851baf8f74a1aa64b94d6e1d1b350.bigdata*********.com   10.49.128.6   80      2m32s

Was this patch authored or co-authored using generative AI tooling?

No.

mridulm · 2024-04-23T15:32:30Z

+CC @zhouyejoe

...etes/core/src/main/scala/org/apache/spark/deploy/k8s/features/DriverServiceFeatureStep.scala

pan3793 · 2024-04-23T18:36:46Z

cc @dongjoon-hyun @yaooqinn @LuciferYang @EnricoMi appreciate it if you could have a look on this idea

dongjoon-hyun · 2024-04-23T20:28:15Z

...etes/core/src/main/scala/org/apache/spark/deploy/k8s/features/DriverServiceFeatureStep.scala

Please don't do that. This is a huge regression for the existing system by reducing the space of service name.

make sense, we can generate the ingress name independently to avoid such regression

...rce-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesConf.scala

dongjoon-hyun · 2024-04-23T20:36:54Z

...etes/core/src/main/scala/org/apache/spark/deploy/k8s/features/DriverIngressFeatureStep.scala

Since this is not a bug, please move this logInfo to outside of this PR with a different message.

dongjoon-hyun · 2024-04-23T20:37:48Z

...etes/core/src/main/scala/org/apache/spark/deploy/k8s/features/DriverIngressFeatureStep.scala

Instead of warning like this, please check at line 37 together and proceed with no-op.

dongjoon-hyun · 2024-04-23T20:39:01Z

...etes/core/src/main/scala/org/apache/spark/deploy/k8s/features/DriverIngressFeatureStep.scala

This return statement implies that we might have too complex code path here.

dongjoon-hyun · 2024-04-23T20:41:04Z

...ernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/KubernetesDriverBuilder.scala

Can we provide a way for Spark Job runs successfully even DriverIngressFeatureStep and its K8s resources fails? IIUC, if something goes wrong, the Spark job is not started, isn't it?

that's a good point.

yes, any failure during resource creation will fail the job submission at the current stage, and I did see the ingress creation failure cases internally. we mitigate it by providing a switch to allow users to disable creating ingress for certain jobs by spark.kubernetes.driver.ingress.enabled.

currently, we create resources in batches,

spark/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/KubernetesClientApplication.scala

Lines 161 to 181 in 48e207f

// Refresh all pre-resources' owner references

try {

addOwnerReference(createdDriverPod, preKubernetesResources)

kubernetesClient.resourceList(preKubernetesResources: _*).forceConflicts().serverSideApply()

} catch {

case NonFatal(e) =>

kubernetesClient.pods().resource(createdDriverPod).delete()

kubernetesClient.resourceList(preKubernetesResources: _*).delete()

throw e

}

// setup resources after pod creation, and refresh all resources' owner references

try {

val otherKubernetesResources = resolvedDriverSpec.driverKubernetesResources ++ Seq(configMap)

addOwnerReference(createdDriverPod, otherKubernetesResources)

kubernetesClient.resourceList(otherKubernetesResources: _*).forceConflicts().serverSideApply()

} catch {

case NonFatal(e) =>

kubernetesClient.pods().resource(createdDriverPod).delete()

throw e

}

if we want to make some resources optional, we need to add a new batch to handle them independently. meanwhile, a new method may need to add in KubernetesFeatureConfigStep, something like

trait KubernetesFeatureConfigStep { ... def getAdditionalOptionalKubernetesResources(): Seq[HasMetadata] = Seq.empty }

@dongjoon-hyun WDYT?

on the other hand, silently ignoring a resource that the user requested may confuse users. for example, if something goes wrong during the spark UI start, we fail the job immediately.

we need a decision here.

option 1: provide a switch spark.kubernetes.driver.ingress.enabled to allow the user to explicitly enable/disable ingress, and fail the job if ingress creation failed.

pros:

behavior consistent with spark.ui.enabled;

if ingress creation failed, the user knows what happened rather than confusing why Spark UI is unavailable;

code change is simple;

cons:

ingress is actually an optional resource, technically, Spark could run well without ingress.

option 2: treat ingress as an optional resource, ignore ingress creation failure
pros:

Keep the same stability for Spark jobs submission even with ingress enabled.

cons:

ingress resource is not reliable even the user enable ingress explicitly

we need to touch the developer interface to achieve such functionalities (there is one proposal above)

internally, we adopted option 1, and some users explicitly disabled the ingress because they didn't want the job submission to fail because of ingress creation failure.

@dongjoon-hyun @yaooqinn @mridulm @LuciferYang please let me know which one you are preferred.

To @pan3793 , I'm very careful about this PR because we consider Having K8s Ingress is more vulnerable than having Spark UI because it increases the boundary of exposure. Spark UI itself is not open to the other pods if you design to isolate Spark pod-to-pod communication carefully.

In addition, some K8s implementations (and their security backend) increases K8s Ingress creation. I expected a few minutes delays for only K8s ingress entity creations. It could mean a delay of Spark Job time too if we design to wait for K8s ingress.

Since we verified that the deletion is handled by K8s ownership correctly. Let's double-check the following.

What is the job start time for Option 1? There is no delay?

If there is a delay, is there a way to mitigate it?

If there is no delay because it is handled separately, what happens during the time gap after the job start already and K8s Ingress is still under construction. Are we in the same situation where Option 2's Cons 1?

I expected a few minutes delays for only K8s ingress entity creations.

that surprises me, in our cluster, the overhead of ingress creation is negligible.

I tested with ingress enabled/disabled 5 times in internal cluster to measure the job submission time by spark.app.startTime - spark.app.submitTime. (based on the implementation of option 1)

ingress enabled: 20715, 24954, 17129, 16631, 16076
ingress disabled: 20246, 20262, 17176, 18633, 19962

the difference is majorly caused by Pod creation, ingress creation is negligible.

if the ingress creation is too heavy, option 2 is the only choice.

dongjoon-hyun · 2024-04-23T20:42:44Z

resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala

I believe we don't need this because KUBERNETES_INGRESS_HOST_PATTERN is mandatory always. Shall we remove this redudant configuration?

dongjoon-hyun · 2024-04-23T20:45:58Z

resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala

May I ask when you want to override this? And, is it safe in terms of the security?

internally, we only use path / with pathType ImplementationSpecific, let me hardcode it and make it configurable if someone has real request in the future

dongjoon-hyun · 2024-04-23T20:53:27Z

...etes/core/src/main/scala/org/apache/spark/deploy/k8s/features/DriverIngressFeatureStep.scala

It seems that there is no owner setting here. Could you double-check if this increase the number of ingress monotonically? I'm wondering who and when this ingress is removed.

yes, OwnerReference is handled at

spark/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/KubernetesClientApplication.scala

Line 175 in 48e207f

addOwnerReference(createdDriverPod, otherKubernetesResources)

I checked that deleting the driver pod removes the ingress too

Thank you for the confirmation.

dongjoon-hyun

Hi, @pan3793 . I left a few comments.

dongjoon-hyun · 2024-04-29T01:51:12Z

Gentle ping, @pan3793 . It would be great if we can land this before Apache Spark 4.0.0-preview.

To @cloud-fan , just a question, when do you think you are going to cut 4.0.0-preview tag?

pan3793 · 2024-04-29T06:36:24Z

@dongjoon-hyun sorry for late, I'm a little busy these days, will address comments soon

This reverts commit b45bb1688e07b1f7a3639c1015a337d3160644b8.

dongjoon-hyun

Also, please address this.

https://github.com/apache/spark/pull/46184/files#r1576870634

We don't need spark.kubernetes.driver.ingress.enabled because spark.kubernetes.driver.ingress.host is the same semantic.

dongjoon-hyun · 2024-05-01T23:25:59Z

For the record, Apache Spark 4.0.0-preview is scheduled on next Monday.

dongjoon-hyun · 2024-05-10T03:44:12Z

Just FYI, please take your time. We can target this for Apache Spark 4.0.0.

github-actions · 2024-08-21T00:21:29Z

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!

melin · 2025-04-23T10:13:47Z

@pan3793 Is there any new progress in this pr?

To access the url: http://hostname:port/proxy/[applicationId], add conf: spark.driver.extraJavaOptions="-Dspark.ui.proxyBase=/proxy/applicationId"

example:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: spark-app-ingress
  namespace: superior
spec:
  ingressClassName: nginx
  rules:
  - host: example.com
    http:
      paths:
      - path: /proxy/spark-1745724044946
        pathType: Prefix
        backend:
          service:
            name: spark-1745724044946-driver-svc
            port:
              number: 4040

To access the url http://hostname:port/proxy/spark-174572404494 is redirected to the http://hostname:port/jobs。 Failed to access the UI.

sparkcontext initializes the ui, and the basepath is empty

@dongjoon-hyun

github-actions bot added the KUBERNETES label Apr 23, 2024

pan3793 marked this pull request as draft April 23, 2024 14:54

pan3793 marked this pull request as ready for review April 23, 2024 18:19

pan3793 commented Apr 23, 2024

View reviewed changes

...etes/core/src/main/scala/org/apache/spark/deploy/k8s/features/DriverServiceFeatureStep.scala Outdated Show resolved Hide resolved

dongjoon-hyun reviewed Apr 23, 2024

View reviewed changes

...rce-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesConf.scala Outdated Show resolved Hide resolved

dongjoon-hyun reviewed Apr 23, 2024

View reviewed changes

...rce-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesConf.scala Outdated Show resolved Hide resolved

dongjoon-hyun reviewed Apr 23, 2024

View reviewed changes

dongjoon-hyun requested changes Apr 23, 2024

View reviewed changes

pan3793 force-pushed the SPARK-47954 branch from e8b4e16 to 2e50aeb Compare April 29, 2024 09:22

pan3793 added 6 commits April 29, 2024 23:41

[SPARK-47954][K8S] Support creating ingress entry for external UI access

4c59f08

configurable path and pathType

dd069a2

Revert "configurable path and pathType"

1680b7d

This reverts commit b45bb1688e07b1f7a3639c1015a337d3160644b8.

MAX_SERVICE_NAME_LENGTH

acc4965

generate ingress name independently

0c2c2c0

warn

4a0fcae

pan3793 force-pushed the SPARK-47954 branch from 2e50aeb to 4a0fcae Compare April 29, 2024 15:42

dongjoon-hyun requested changes Apr 29, 2024

View reviewed changes

github-actions bot added the Stale label Aug 21, 2024

github-actions bot closed this Aug 22, 2024

	// Refresh all pre-resources' owner references
	try {
	addOwnerReference(createdDriverPod, preKubernetesResources)
	kubernetesClient.resourceList(preKubernetesResources: _*).forceConflicts().serverSideApply()
	} catch {
	case NonFatal(e) =>
	kubernetesClient.pods().resource(createdDriverPod).delete()
	kubernetesClient.resourceList(preKubernetesResources: _*).delete()
	throw e
	}

	// setup resources after pod creation, and refresh all resources' owner references
	try {
	val otherKubernetesResources = resolvedDriverSpec.driverKubernetesResources ++ Seq(configMap)
	addOwnerReference(createdDriverPod, otherKubernetesResources)
	kubernetesClient.resourceList(otherKubernetesResources: _*).forceConflicts().serverSideApply()
	} catch {
	case NonFatal(e) =>
	kubernetesClient.pods().resource(createdDriverPod).delete()
	throw e
	}

[SPARK-47954][K8S] Support creating ingress entry for external UI access #46184

[SPARK-47954][K8S] Support creating ingress entry for external UI access #46184

Uh oh!

Conversation

pan3793 commented Apr 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

mridulm commented Apr 23, 2024

Uh oh!

Uh oh!

pan3793 commented Apr 23, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pan3793 Apr 24, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pan3793 Apr 24, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pan3793 Apr 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pan3793 Apr 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pan3793 Apr 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Apr 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pan3793 commented Apr 29, 2024

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented May 1, 2024

Uh oh!

dongjoon-hyun commented May 10, 2024

Uh oh!

github-actions bot commented Aug 21, 2024

Uh oh!

pan3793 commented Apr 23, 2024 •

edited

Loading

pan3793 Apr 24, 2024 •

edited

Loading

pan3793 Apr 24, 2024 •

edited

Loading

pan3793 Apr 29, 2024 •

edited

Loading

pan3793 Apr 30, 2024 •

edited

Loading

pan3793 Apr 29, 2024 •

edited

Loading

dongjoon-hyun commented Apr 29, 2024 •

edited

Loading

melin commented Apr 23, 2025 •

edited

Loading