Allow injection of arbitrary application secrets into driver/executor Pods

[liyinan926]

Spark applications running in Kubernetes may need application credentials to access certain services, for example, to read data from or write data to database or storage services. As an example, for an Spark application to access Google Cloud Storage (GCS) or Cloud BigQuery through the GCS or BigQuery connectors, a GCP service account key with the right permissions granted must be injected and made available to the connectors. GCP service account keys are certainly not the only type of credentials. Other types of credentials may be needed to access other types of services outside GCP. So we need a generic way of injecting arbitrary application credentials in the form of Kubernetes secrets into the driver/executor Pods.

[mccheah]

Can spark.files be used to inject your secrets? See #393

[erikerlandson]

Could this be generalized to include configmap injections?

[erikerlandson]

Also, secrets and configmaps can be exposed via env-vars. Is this a useful alternative mechanism for #395 ?

[liyinan926]

I think the implementation in #395 should be preferred because 1) it feels abusing secrets just for setting env variables, which can be done directly at Pod creation time, and 2) Spark already has native support for setting executor env-vars through the spark.executorEnv. property prefix so we should leverage that.

[luck02]

I've been using podpresets:

kind: PodPreset
apiVersion: settings.k8s.io/v1alpha1
metadata:
  name: app-env-presets
  namespace: default
spec:
  selector:
    matchLabels:
      role: appRoleExample
  env:
    - name: exampleEnvVar1
      value: valueEnvVar1
    - name: etcEnv
      value: etcEnvValue

Docs:
https://kubernetes.io/docs/tasks/inject-data-application/podpreset/

Needs admission controller configs set:

minikube start --extra-config=apiserver.Admission.PluginNames=PodPreset,ServiceAccount

PodPreset is the one you need for PodPresets.

I haven't quite gotten this into production, prod admissionController configs are different.

I needed ServiceAccount as well with minikube v.021, but not specifically for podPresets.

⇒  minikube version
minikube version: v0.21.0

Be aware that the API for plugins / AdmissionControllers changed as of v.021 minikube. It used to look like this:

minikube start --extra-config=apiserver.GenericServerRunOptions.AdmissionControl=PodPreset

You need to assign a matching label in your spark-submit call like:

      --conf spark.kubernetes.driver.labels=role=appRoleExample \
      --conf spark.kubernetes.executor.labels=role=appRoleExample \

I haven't tried yet, but there's no reason why you shouldn't be able to use secrets...

[luck02]

On futher reflection, podPresets are still alpha. Not sure if / when they'll go to beta: #38

However, they sure would solve allot of use cases.

Does anyone know when we'll see these in beta? I've been using podPresets quite effectively for env vars, it seems like it's purpose built for the use case.

[liyinan926]

One problem with PodPreset is it applies to all Pods with matching labels in a namespace. If a user needs to inject secrets not included in the PodPreset, she's out of luck. Another problem with PodPreset is if the admission controller fails to apply some PodPreset to a Pod, e.g., due to conflicts, it still allows the Pod to be created without having the PodPreset applied. This might cause an affected Spark application to exhibit unexpected behavior instead of failing fast.

[luck02]

If a user needs to inject secrets not included in the PodPreset, she's out of luck

@liyinan926 I'm not sure if that's a problem. My use case envisions strictly the case where I want to provision env / secrets to env for an executing spark job. If I needed to inject secrets not included in a pod preset could I not make another pod preset that applied to a specific label?.

I'd be at least as concerned with having multiple tools that do almost the same job. Code sprawl and confusion and all that.

Another problem with PodPreset is if the admission controller fails to apply some PodPreset to a Pod, e.g., due to conflicts, it still allows the Pod to be created without having the PodPreset applied

This is a legitimate concern, given the choice between two separate but similar mechanisms I'd lobby for one mechanism. I was just doing some research on pod presets. Their still somewhat up in the air. In an ideal world I'd want a mechanism to flag apply failures.

There seems to be allot of discussion on the subject. Hopefully there'll be more clarity soon.
kubernetes-retired/service-catalog#1041 (comment)

[luck02]

Given that's it's still alpha things could definitely move around. One of the things that could be done would be to inspect if podPresets are mapped. You get this podpreset.admission.kubernetes.io/podpreset-podprset-env: 99999 applied to the pod. https://kubernetes.io/docs/tasks/inject-data-application/podpreset/#behavior I'm wondering if you'd want to check for that label and see if it happened. also, if there were other labels applied if something went wrong.