Discuss naming schemes for pods, config maps, and secrets that are created for the job

[mccheah]

Currently we allow setting spark.app.name which will set the application name that the other resource names are derived from. However, we allow setting the driver pod's name directly, but if the driver pod's name is set, the executor pod names are still derived from the Spark application name. This could make naming inconsistent, as observed in #318.

Separately, however, we could take a step back and consider if we should be setting the specific names of each of the components in the system. That is, should we allow setting the specific name of the driver pod in general? One argument against this could be that we are actually similar in precedent to other controller-like entities in the Kubernetes ecosystem, such as ReplicaSets and Deployments. These other higher-order API objects which construct children pods only allow the children's names to be derived from the parent. Or to put it another way, the children pod names cannot be specified directly, but can only be derived from the parent Deployment or ReplicaSet name.

One could propose that when we roll out the CustomResource - that is, the SparkJob object, that this should:

1. Serve as the equivalent of the parent Kubernetes entity,
2. The user should only be specifying the name of this parent resource, and
3. The names of the children would be derived from the parent.

The downside is that this imposes the requirement of having the CustomResource type set up on their cluster, which is a dependency that shouldn't truly be mandatory.

We could remove the API to set the driver pod name directly, which forces everything to be derived from the application name, and thus everything would always be consistent. If the user has the CustomResource type configured, then we would set the CustomResource's name to equal the application name, which makes the derivation and the tree clear as is discussed above. However, users have shown interest in setting this pod's name specifically before (see #258).

[mccheah]

@hustcat - I'm curious to know your use case for #258. In what scenario was it important that the driver pod's name be given a particular value?

[erikerlandson]

I think removing control of pod name is plausible. That may accentuate the need for users (and apps) to be able to do discovery - figure out which of some set of running spark apps they want. Maybe the ability to add labels meets that requirement?

[mccheah]

Users already have the ability to add custom labels to the driver and executors. I would expect most discovery mechanisms to use those.

[foxish]

I agree with this in the context of naming every component, except the driver, (or whatever top-level resource we add, like the CustomResource). There is value in deterministically naming that top-level component, because the name acts as a lock, and sometimes you want to ensure there is exactly one of that component running without having to run discovery.

[mccheah]

I agree with this in the context of naming every component, except the driver, (or whatever top-level resource we add, like the CustomResource). There is value in deterministically naming that top-level component, because the name acts as a lock, and sometimes you want to ensure there is exactly one of that component running without having to run discovery.

Maybe the application name should itself always be the driver pod name then, and we shouldn't be trying to generate the unique identifier with the timestamp for the user?

[foxish]

Yeah, that sounds reasonable to me, if supplemented with an option to generate a name also. Kubernetes components do this using GenerateName.

[duyanghao]

@mccheah we can use spark.kubernetes.driver.pod.name par as the name of driver pod name as well as label:spark-app-id value of both driver and executor pods,in this way,if i set par --conf spark.kubernetes.driver.pod.name=xxx,then the driver pod name will be xxx with label:spark-app-id=xxx,and executors pod name will be xxx-exec-1 with label:spark-app-id=xxx.
For examples,if i submit with pars:

--conf spark.kubernetes.driver.pod.name=spark-pi-today-a
--conf spark.executor.cores=2

then driver pod name and labels will be:

kubectl describe pods/spark-pi-today-a
Name:		spark-pi-today-a
Namespace:	default
Node:		xxx
Start Time:	Tue, 06 Jun 2017 11:12:01 +0800
Labels:		spark-app-id=spark-pi-today-a
		spark-app-name=spark-pi
		spark-role=driver
Status:		Succeeded

executor #1 pod name and labels will be:

kubectl describe pods/spark-pi-today-a-exec-1
Name:		spark-pi-today-a-exec-1
Namespace:	default
Node:		xxx
Start Time:	Tue, 06 Jun 2017 11:12:33 +0800
Labels:		spark-app-id=spark-pi-today-a
		spark-exec-id=1
		spark-role=executor
Status:		Succeeded

executor #2 pod name and labels will be:

kubectl describe pods/spark-pi-today-a-exec-2
Name:		spark-pi-today-a-exec-2
Namespace:	default
Node:		xxx
Start Time:	Tue, 06 Jun 2017 11:12:33 +0800
Labels:		spark-app-id=spark-pi-today-a
		spark-exec-id=2
		spark-role=executor
Status:		Succeeded

the main reason for this is that we can search all pods for a spark application with the specified label(spark-app-id) value through par spark.kubernetes.driver.pod.name,which can be more easily combined with web server(if there is any),guess that use A specifies par --conf spark.kubernetes.driver.pod.name=xxx,then web server could get all pods information through k8s with label:spark-app-id=xxx,that is good anyway!!!

And I have created a PR to do this.Take a look at here.

[mccheah]

@duyanghao can you achieve a similar goal by setting a value in spark.kubernetes.driver.labels and spark.kubernetes.executor.labels, such that they both have a label with the same key and same value?

bin/spark-submit --conf spark.kubernetes.driver.labels=mylabel1=mylabel1value --conf spark.kubernetes.executor.labels=mylabel1=mylabel1value

Then you can search for all pods with mylabel=mylabel1value.