Discuss how to improve Spark-on-K8s developer workflow

[kimoonkim]

@foxish @varunkatta

We had a discussion on how to improve our workflow. In particular, how we can shorten the 1. maven/sbt build time and 2. docker image build/push time.

I was chatting a bit more on this with @varunkatta. He suggested one could only build k8s module jars and update a distribution with the new jars. @varunkatta Can you comment on how you did this?

It occurred to me that we could extend this idea to docker image building. We could have a base image that has spark core jars, but not k8s module jars. Then the driver and executor images could extend from the base image by adding k8s module jars. Most of the time when we modified our code, we'll only build the child docker images. Hopefully, this will shorten the build/push time.

What do you think?

[varunkatta]

This is what I did at a high-level. Built kubernetes spark core, and updated the jar which is present in prebuilt tar zipped distribution with the new updated jar freshly built. Steps outline below.

# assumes you are in top level project dir.
./build/mvn -pl resource-managers/kubernetes/core -Pkubernetes -am package -DskipTests;
# assumes you made a distribution named spark-2.1.0-k8s-0.1.0-SNAPSHOT-bin-k8s-20170301.tgz
tar -zxf spark-2.1.0-k8s-0.1.0-SNAPSHOT-bin-k8s-20170301.tgz; 
scp resource-managers/kubernetes/core/target/spark-kubernetes_2.11-2.1.0-k8s-0.1.0-SNAPSHOT.jar spark-2.1.0-k8s-0.1.0-SNAPSHOT-bin-k8s-20170301/jars
tar -zcf spark-2.1.0-k8s-0.1.0-SNAPSHOT-bin-k8s-20170301.tgz spark-2.1.0-k8s-0.1.0-SNAPSHOT-bin-k8s-20170301;

[varunkatta]

I like the idea of having a base docker with all the required spark jars except the spark-kubernetes jar, and incrementally building the new docker image with the freshly baked spark-kubernetes jar. Curious to see how many seconds it will shave off of current docker image building time.

[mccheah]

@kimoonkim or @varunkatta - can either of you ship a PR to use base docker images? Keep in mind to also update SparkDockerImageBuilder in the integration test package also.

One question though is what tag of the base image the child images should depend on. The child dockerfiles reference the parent image by image-name:image-tag, e.g. spark-base:latest or spark-base:2.1. I'm not sure what the right version for the children to depend on is here. Whatever that version is, the SparkDockerImageBuilder in the integration test project needs to tag the base image accordingly.

[varunkatta]

I will send a send a PR for this. I am thinking we should use spark-base:latest over using the version number in the tag.

[varunkatta]

There is PR in the Kubernetes ansible project for this - apache-spark-on-k8s/ansible#4