Static Allocation MVP umbrella issue

[foxish]

I'm collecting here the various issues I find as I go over the prototype. Folks can vet that it actually is an issue, and then pick it up.
If you're working on it, I'd suggest that you write your name next to it.

Bugs:

• Cleanup the driver, secret and service after the SparkJob completes (Flag for preserving driver pods that failed to start #17)

Usability:

• k8s:// as synonym for k8s://https:// (k8s:// as synonym for k8s://https:// #19)
• Getting spark driver output and inspecting logs using kubectl. (inherit IO to get kubectl logs output #13)
• Executor pods should be named according to the name of the spark job itself, rather than exec style of naming we currently have (Added service name as prefix to executor pods #14).
• Driver pod must complete successfully once SparkJob is done.

Misc

• Future errors out after timeout of 30s: is this appropriate? I've seen several driver pods take longer (especially on first launch). What's an appropriate timeout?
• Testing using GKE clusters. [@foxish]

[foxish]

cc @mccheah @erikerlandson @ash211 @iyanuobidele @ssuchter

[foxish]

The steps I follow to run the prototype are:

1. Create the distribution using:

./dev/make-distribution.sh -Pkubernetes -DskipTests -Darguments="-DskipTests"

2. Create new driver/executor images using the $SPARK_HOME/dist directory created.
3. Run a simple job:

./bin/spark-submit 
    --deploy-mode cluster 
    --class org.apache.spark.examples.SparkPi 
    --master k8s://https://104.154.43.148  
    --conf spark.executor.instances=5 
    --conf spark.app.name=foxish 
    --conf spark.kubernetes.namespace=default 
    --conf spark.kubernetes.driver.docker.image=foxish/spark-driver:0.2 
    --conf spark.kubernetes.executor.docker.image=foxish/spark-executor:0.2 
    /usr/local/original-spark-examples_2.11-2.2.0-SNAPSHOT.jar 10000

[ash211]

@mccheah and I just ran through this codebase on an internal k8s cluster and got things all running. A couple bits came out of this that are relevant for the group:

• should put Dockerfiles into dist/ folder so don't have to copy them there when building images
• consider creating a shell script that builds and tags the docker files?
• the driver pod doesn't shut itself down on completion
• the driver pod doesn't delete its service entry on completion

Additionally:

• we set up and ran a private docker registry, and used docker images from there for the job
• we needed to grab the apiserver cert and use spark.kubernetes.submit.caCertFile for submission, so verified that config works

[foxish]

Thanks @ash211 and @mccheah for confirming those issues. I can help fix some of them. I am running one set of tests across different k8s versions as well.
I'm not sure why the apiserver cert had to be passed, since the pods mount it and are capable of communicating with the apiserver directly from within the cluster.

On the first issue, I'm still having trouble getting it to actually work on a cluster established on GCE running built from HEAD on kubernetes/kubernetes. I suspect it may be something to do with RBAC or some authorization plugin. I'm still investigating and will update the issue when I discover the cause.

[ash211]

The reason we needed to pass the apiserver cert is because we had a cluster running out on private infra with a non-CA SSL cert, and my laptop didn't have the apiserver's custom cert loaded into the system keystore. Running services with non-CA SSL certs is actually very common for us at Palantir so it was good for us to verify it works.

No idea what your GCE issue is with HEAD on kubernetes/kubernetes -- good luck debugging! We were using kubernetes 1.5.1 and Docker 1.12.3

[foxish]

Turned out to be a regression at our end since after 1.5. Fix in progress, the prototype looks good.
Sorry about the confusion.

Works as expected on kubernetes 1.4.x as well as 1.5.x.

[ash211]

@foxish can you take this one? "Support for RBAC authentication which is default v1.6 onwards"

I think you're one of the more familiar folks with the Kubernetes side right now, and I'm not quite sure what the problem here is. Especially since 1.6 is not yet released :)

[foxish]

Yes, was planning on doing that one. I'm waiting for the code to stabilize in the Kubernetes repo. In all likelihood, we'll include credentials when we send jars from the user's local disk to the rest submission server. With rbac enabled, anonymous access to the service via the apiserver's service/proxy endpoint will no longer be allowed.

[foxish]

I'm punting the RBAC issue to later, and for beta, since it hasn't become the default yet.