Cutting the Spark 2.2 release

[foxish]

We can cut this and mark it as beta for now. I don't think there is a need to update documentation yet however to point to 2.2. This is a prerequisite to announcing upstream.
Steps:

1. Build a distribution with and without hadoop
2. Tag the commit, draft a github release, and upload the tar.gz files in (1)
3. Build and upload the images (appropriately tagged) to the kubespark repo
4. Fill in the release description on GitHub

@apache-spark-on-k8s/contributors

"blocking" PRs:

1. Including documentation for PySpark Support userdocs#8
2. PySpark Submission Failing on --py-files #407
3. An entrypoint.sh script to add a passwd entry if one does not exist #404
4. Initial architecture documentation. #401
5. Allow configuration to set environment variables on driver and executor #424
6. Flag-guard expensive DNS lookup of cluster node full names, part of HDFS locality support #412

[erikerlandson]

I'm eager to announce, but I'm expecting the python support to be a popular feature, and I'm still hoping for at least a quick-start example for people to refer to. Otherwise they'll have nothing to refer to, IIUC.

[foxish]

+1, @ifilonenko, could you please prioritize the docs?

[erikerlandson]

@foxish the docs look good to me: apache-spark-on-k8s/userdocs#8
If #407 passes, we should include that.

[erikerlandson]

I'd also like to merge #404 before cutting, unless there are any objections.

[foxish]

+1, makes sense to me @erikerlandson. Let's merge those and get out the release. We might have to cut another 2.1 as well because of the recently discovered pyspark bug.

[erikerlandson]

@liyinan926 for the sake of consistency, can you paste the dev/make-distribution and build/mvn commands you used to do the latest 2.1 build here? I'd like to make sure I'm using the same build options, to the extent possible

[erikerlandson]

Recording my build commands here.

dist tarball:

$ ./dev/make-distribution.sh --pip --tgz -Pmesos -Pyarn -Pkinesis-asl -Phive -Phive-thriftserver -Pkubernetes -Phadoop-2.7 -Dhadoop.version=2.7.3
$ ./dev/make-distribution.sh --pip --tgz -Pmesos -Pyarn -Pkinesis-asl -Phive -Phive-thriftserver -Pkubernetes -Phadoop-2.7 -Dhadoop.version=2.7.3 -Phadoop-provided

image builds (from spark-2.2.0-k8s-0.3.0-hadoop-2.7.3.tgz):

$ sed -i 's/FROM spark-base/FROM kubespark\/spark-base:v2.2.0-kubernetes-0.3.0/' dockerfiles/*/Dockerfile
$ docker build -t kubespark/spark-base:v2.2.0-kubernetes-0.3.0 -f dockerfiles/spark-base/Dockerfile .
$ docker build -t kubespark/spark-driver:v2.2.0-kubernetes-0.3.0 -f dockerfiles/driver/Dockerfile .
$ docker build -t kubespark/driver-py:v2.2.0-kubernetes-0.3.0 -f dockerfiles/driver-py/Dockerfile .
$ docker build -t kubespark/spark-executor:v2.2.0-kubernetes-0.3.0 -f dockerfiles/executor/Dockerfile .
$ docker build -t kubespark/executor-py:v2.2.0-kubernetes-0.3.0 -f dockerfiles/executor-py/Dockerfile .
$ docker build -t kubespark/spark-init:v2.2.0-kubernetes-0.3.0 -f dockerfiles/init-container/Dockerfile .
$ docker build -t kubespark/spark-resource-staging-server:v2.2.0-kubernetes-0.3.0 -f dockerfiles/resource-staging-server/Dockerfile .
$ docker build -t kubespark/spark-shuffle:v2.2.0-kubernetes-0.3.0 -f dockerfiles/shuffle-service/Dockerfile .

[liyinan926]

@erikerlandson I used the following commands:

dev/make-distribution.sh --tgz -Phadoop-2.7 -Pkubernetes
dev/make-distribution.sh --tgz -Pkubernetes

[erikerlandson]

@liyinan926 thx - my reading of the doc says that not providing the -Phadoop-x.x just defaults it to hadoop-2.2.0, should we be using -Phadoop-provided for the "no-hadoop" version?

Also, are you building the docker images from the (unpacked) tarballs?

cc @foxish

[foxish]

Good catch @erikerlandson. I had missed that in the documentation as well.
We should use the -Phadoop-provided like you said.

[liyinan926]

Yes, I used the unpacked tarball (with hadoop) to build and push the images.

[liyinan926]

@foxish @erikerlandson I will rebuild the no-hadoop distribution and upload it with -Phadoop-provided.

[erikerlandson]

@apache-spark-on-k8s/contributors I have cut tarballs and images for v2.2.0-kubernetes-0.3.0 - if somebody can smoke test these as a sanity check it would be great!

[erikerlandson]

2.2 is cut and announced