Support creating Ingress services for driver

[ash211]

Access to the driver is needed for at least 3 purposes:

• upload local jar resources to the driver pod for execution
• access Spark UI
• additional ports on the driver for interactive applications

External access to the driver for jar upload is currently run through a NodePort proxy (from the apiserver proxy recently), but for some workflows @mccheah and I have we will need to upload the jars through an Ingress-configured service instead.

The precise details are currently unclear, but over the next few weeks I expect we'll gain more clarity on the need and eventually be submitting a PR.

[mccheah]

It is non-trivial to introduce this feature in a way that is flexible enough to be used in every Kubernetes cluster and with every Ingress controller. For example, a first attempt proposed in #116 created a new Ingress resource for every application, but such an approach would be expensive if used with GKE and GCE.

My proposal is the following:

• Support spark.kubernetes.driverIngressName as a String configuration
• The Client checks to see if the Ingress with the given name exists in the namespace set by spark.kubernetes.namespace
• If the Ingress does not exist, create it with some default settings (this is debatable since the default settings are difficult to determine)
• Patch the Ingress to add a new rule to route to the new driver service

One of the problems with the above proposal is determining when to clean up the Ingress object. We also would have to make sure the path that are added to the Ingress are deleted when the Spark application completes.

A second alternative is to not create the Ingress with Spark at all. Instead, the Spark application's third party resource can include a field for an external URL that is filled in by a controller in the Kubernetes cluster. The controller can listen for the creation of the Spark application and fill in the external URL field. The controller can choose the URL with an Ingress, or with NodePort or some other externally-routable address.

@foxish @ash211 @iyanuobidele @erikerlandson thoughts on these options or any other options?

[foxish]

If the Ingress does not exist, create it with some default settings (this is debatable since the default settings are difficult to determine)

I think there is no universally applicable default, because specific ingress controllers can take different annotations.
The user would have to supply a YAML file which contains the entire spec so as to ensure it supports every use case.

One of the problems with the above proposal is determining when to clean up the Ingress object

I don't think there is a way in which we can have rules automatically expire, or be garbage collected.
One may need a different controller cleaning up after the ingress resource.

Like we discussed earlier, I like the second alternative a little better in general, regardless of whether we decide to support ingress natively or not:

• It decouples the URIs themselves from the mechanism by which we expose them: nodeport services, ingresses, etc would all fill in the driverSubmissionExternalUrl field which spark-submit will submit to.
• It fits in well with the use case where the cluster can run a custom controller that creates an ingress resource/performs some other action to expose the driver pod's ports, and eventually sticks the externally accessible path into the TPR.

The side effect of the above may be that we make the TPR an essential part of the solution, which it was not so far. If that is not desirable, we may be able to also get away with setting the driverSubmissionExternalUrl field as an annotation on the driver pod.

[mccheah]

I don't there is a way in which we can have rules automatically expire, or be garbage collected.
One may need a different controller cleaning up after the ingress resource.

The driver pod itself could adjust its own Ingress, perhaps in the stop() method of the scheduler backend which should be called when the driver process shuts down.

The main concern I have with the second solution is what it will take for users to install cluster-side. The first solution requires no additional setup cluster-side apart from the presence of any Ingress controller to which Ingresses can be created, and administered. The second solution requires another component to be implemented and installed on the cluster that provides the external URIs, which seems superfluous given that the Ingress API itself seems to be designed for that use case precisely. If the Ingress API exists, shouldn't Spark be using it?

[foxish]

The driver pod itself could adjust its own Ingress, perhaps in the stop() method of the scheduler backend which should be called when the driver process shuts down.

This is only true under the case of graceful termination. A node failure for example, would not allow enough time for the driver pod to clean up after itself. The primary reason for implementing the garbage collection using OwnerRefs was that, so as to ensure we do not leak resources.

[ash211]

In general, it seems reasonable to use both self-cleanup and OwnerRefs. The first gets prompt cleanup, the second gets eventual cleanup on failure. @foxish Are you suggesting that OwnerRefs can't be relied on for eventual cleanup of a "shared" Ingress, where the cleanup is to modify the Ingress rather than delete it?

[foxish]

Are you suggesting that OwnerRefs can't be relied on for eventual cleanup of a "shared" Ingress, where the cleanup is to modify the Ingress rather than delete it?

Yes, the OwnerRefs and GarbageCollection can only perform a cleanup at the resource level, and not at the subresource/field level. So, under this model, we'd need a custom controller to cleanup fields in the ingress resource anyway.

[foxish]

To clarify, by cleanup, I mean deletion of a resource itself. OwnerReferences do not currently support any way to plug in custom logic.

[mccheah]

One idea that was discussed offline was the following:

• Expose a Spark boolean configuration called spark.kubernetes.driver.useExternalUriProvider
• If the above configuration is set to false, submit the driver using the NodePort service as currently implemented
• Otherwise, still create the service and pod but place an annotation on the service - say spark/provideExternalIp. The service would be created with type ClusterIP.
• Some external component that watches the API server can listen to the creation of services created with the annotation spark/provideExternalIp. If such a service is created, this external component would be expected to patch the service with another annotation: spark/resolvedExternalIp. The value of this annotation is the external URI that should be used by the client to route the application submission request to the driver pod.
• The submission client waits for the service to have the given annotation be filled in, and uses the value of that annotation as the external URI to route to the driver.

This can all be done without any dependency on the third party resource. The user does have to provide their own implementation of the external component, but in this case the API for the driver to receive an externally-routable URI is well-defined.

[mccheah]

The above conclusions are quite different from this issue's title and description. I'm in favor of closing this and opening a new issue, something along the lines of "Support providing the driver URI from an external controller". Thoughts @ash211 @foxish ?

[foxish]

SGTM