Replace usage of API server proxy in Client with Ingress or NodePort

[mccheah]

Currently the K8s client running on the user's machine contacts the REST server to submit local files and other configurations. The Client uses the API server proxy to reach the service endpoint. We could think about whether or not we want to replace the usage of the proxy with either an Ingress proxy or exposing the service on a node port. The tradeoffs between these three different alternatives should be weighed before we move forward with one of them.

[mccheah]

@foxish @erikerlandson what are your thoughts on the different options here?

[foxish]

There are trade-offs here. The APIServer proxy won't scale and will overload the master, so it would be best to move away from it for submitting large files. The LoadBalancer and Ingress resources are not necessarily supported on all clusters, so that makes them a little less preferable in my opinion. NodePort to me seems like a viable alternative. However, it has the unfortunate side effect of exposing one port on all nodes and placing an upper limit on the number of running drivers in a cluster. That might not really be an issue however. We would need to figure out authentication ourselves in that case, and a way to shut down and stop listening on that NodePort as soon as file submission completes.

Do we have an idea of how large the files submitted by the client may be?

[mccheah]

As large as it would take to store an entire application's custom jars. We do gzip compress the shipped contents, but I agree we would want something that scales better.

Is there a timeline for supporting Ingress everywhere and having it on by default?

Edit: Does the story change if we also say we want to SSL encrypt the shipped jars? I would imagine it would be trickier to get this right with the API server proxy as well. I'm not sure if it makes a difference for either the Ingress or the NodePort approach. Ingress does support SSL termination but it would be nice to get the SSL encryption end to end all the way to the driver pod itself.

[foxish]

I'm going to look at the networking SIG's mandate. I'll get back to this issue once I talk to folks from the SIG.
Submitting local jars is just one possible mechanism, and I expect that baking the jars into the image, or using HDFS/network storage to store jars will be the major use case.

If we are going to use authentication and nodeports, I'm wondering if it might make sense to have one submission server per cluster (rather than per driver), like the model that https://github.com/spark-jobserver/spark-jobserver uses. Thoughts?

[mccheah]

Having one submission server for the whole cluster could also work, but it would require the extra deployment step to set it up on the user's part. Having the spark-submit procedure set up everything automatically I think would be a more ideal experience.

What have been some of the usual ways that SSL has been set up on Kubernetes services in general?

[foxish]

Having one submission server for the whole cluster could also work, but it would require the extra deployment step to set it up on the user's part. Having the spark-submit procedure set up everything automatically I think would be a more ideal experience.

I agree that it would be more seamless. However, if we do require the submission server to exist, it would need to be setup once by the cluster administrator, which is not necessarily bad. I think Mesos on Spark uses this model.

SSL setup: using the nginx ssl proxy https://github.com/GoogleCloudPlatform/nginx-ssl-proxy is a common way to go about that.

[mccheah]

Actually I'm not sure if the singleton submission server works in our case. The reason is that we're trying to deploy the user's jars onto the local disk of the driver that ends up running. If the driver runs in its own pod, the submission server needs to be running with a disk shared to the driver pod. But of course since each driver runs in its own pod, that means each driver needs to have its own submission server.

[foxish]

I was expecting that such a singleton submission server would work exactly the same as the APIServer proxy, proxying content to each possible driver pod.

[mccheah]

The driver pod would still need to have a server to receive the content. In such a case I don't see what the singleton submission server which is acting as a proxy would do that cannot be accomplished by using Kubernetes primitives such as services and Ingress. Is there an advantage to using our own proxy server as opposed to leveraging what Kubernetes already provides?

[foxish]

The driver pod would still need to have a server to receive the content.

Yes, but that IP:port would not need to be exposed outside the cluster. Ingress and LoadBalancer currently work as abstractions over NodePort and need specific cloud provider support to work. So, the question is whether we want one NodePort allocated per driver, or one NodePort per cluster for a Spark submission proxy. I summarized the two approaches below.

1. One submission service per driver (current model)

   • Simpler as we are launching the server and client with each spark-submit
   • We need as many NodePorts to be opened as there are SparkJobs uploading local JARs.
   • By default, there are only around 2700 ports that can be used by default, and they're a cluster-wide resource.
   • We do not want to impact multi-tenancy adversely by occupying too many NodePorts, because it is a cluster-wide resource.

2. One submission service for the entire cluster

   • We need exactly one NodePort to be exposed outside the cluster.
   • This proxy would need to authenticate on behalf of the user and proxy files to any driver pod.
   • We can scale this proxy up/down independently depending on the load

I think we should pick according to how long we expect the NodePort service to live. If we ensure that the lifetime of the service is very short, we can do one NodePort per driver. There may be performance implications of having a large number of services (1000s), but we haven't done a scalability test yet to confirm.

[foxish]

I had a discussion with some networking folks. There are no scalability limits that we know of, so, we should be okay with one NodePort per driver, as long as we return it back to the allocator pool as soon as uploading of local jars is complete. Keeping the service around for the entirety of the driver's lifetime would definitely make us hit NodePort limits rather quickly.

[mccheah]

The NodePort for the submission would be fairly short lived as it only needs to upload the data to the driver. We can open the port for the duration of the submission and then close it when we're done. This wouldn't scale if someone submits a large number of Spark jobs in quick succession though, causing the port space to run out.

There is also something to be said about the Spark UI, which is long-lived. We will probably want to have a proxy server for that, perhaps as part of the work we're doing around creating Spark-based constructs in core Kubernetes, etc.

[mccheah]

The more I'm thinking about this though the more I'm liking the idea of building our own proxy service, because of the use case of burst-submits of jobs which I think may actually be quite common. It could be fairly lightweight as well and doesn't actually have to handle much of the authentication matters since this server wouldn't be responsible for the creation of any Kubernetes resources but would just be handed a pod/service IP to forward the submission request to, as well as the certificate that the driver generated for that submission.

Edit: It would be nice to use something simple like Nginx but unfortunately I think the two hops of SSL will make it difficult, particularly since all of the drivers would ideally have unique credentials.

[foxish]

We should have around 2000 ports available for use by default (and can be raised to 10,000 or higher using an argument to the master). For a burst submission to use up all of those ports, all of those jobs would need to be trying to upload jars concurrently. Do you see that being a likely scenario? From talking to people, it doesn't seem like that would be common.

[mccheah]

Hm, yeah perhaps it won't be that common.

We would probably need to build a proxy server to handle the Spark UI case anyways. Perhaps we can revisit the discussion of proxying the file uploads through that when we build that out. For V0 I think NodePort is sufficient.