Support setting ImagePullSecrets on the Spark pods

[ash211]

For connecting to an authenticated docker registry where the underlying k8s cluster doesn't already have those docker creds, k8s supports specifying that registry's keys in a k8s secret and referring to the secret in the podspec. We should support that in spark-on-k8s

Following docs at https://kubernetes.io/docs/user-guide/images/#specifying-imagepullsecrets-on-a-pod that looks like:

Create a secret:

• $ kubectl create secret docker-registry myregistrykey --docker-server=DOCKER_REGISTRY_SERVER --docker-username=DOCKER_USER --docker-password=DOCKER_PASSWORD --docker-email=DOCKER_EMAIL

Use the secret on a pod using imagePullSecrets:

apiVersion: v1
kind: Pod
metadata:
  name: foo
  namespace: awesomeapps
spec:
  containers:
    - name: foo
      image: janedoe/awesomeapp:v1
  imagePullSecrets:
    - name: myregistrykey

We need a way to specify the creds that are used to pull the Spark images from a docker registry.

This is probably additional support for using fully-customizable YAML for pod specs: #38

[iyanuobidele]

We had this fix from the fork we worked on before. On @foxish's branch.

I think this should go in the MVP. Since its a fairly simple fix.
Also, we can assume people will build their own flavors of the spark driver/executor images for different reasons and push to a private registry.

We essentially just used a spark conf property and added it to the pod definitions if set. spark.kubernetes.imagePullSecret

Thoughts?

[ash211]

Hmm, alternatively I was able to set an imagePullSecret on my service account and use that to resolve docker images. That seems like better practice than specifying different imagePullSecrets per job anyway, so I'm not sure this is still necessary.

https://kubernetes.io/docs/user-guide/service-accounts/#adding-imagepullsecrets-to-a-service-account

Is there a need for creating spark.kubernetes.imagePullSecret functionality that wouldn't be covered by adding imagePullSecrets to the service account instead?

[iyanuobidele]

No. That's definitely much cleaner.

[ash211]

In that case let's hold off on adding additional complexity until someone needs it and there's no workaround. Setting secrets on the service account seems like much better practice so I'd like to encourage folks to use that instead.