Support non-service account based authentication in the driver pod #182
Description
Currently we make the driver pod use the service account token that is mounted by Kubernetes into /var/run/secrets/kubernetes.io/serviceaccount/token as well as the certificate file under /var/run/secrets/kubernetes.io/serviceaccount/ca.crt. However, some use cases might warrant authentication via user-provided OAuth tokens instead of relying on a service account token.
The proposal is to support a second mode of operation where the driver pod is provided a secret volume containing all of the data provided through spark.kubernetes.submit.clientCertFile, spark.kubernetes.submit.clientKeyFile, etc. In such a mode, the driver pod authenticates to the Kubernetes API server in the same way the driver launcher process itself does.
An extension of this is to allow the driver pod to use distinct credentials from the driver submission client. The driver submission client currently requires the ability to create and access secrets, create+delete services, and create+delete pods. The driver however has no need for permissions to handle the secrets and the services in the namespace, so presumably the driver can use credentials with the appropriate restricted set of permissions. We can default to making the credentials set identical between the driver submission and the driver itself, resolve the overridden credentials that the driver pod should use, and mount the secret volume with the resolved credentials.