Cut a new 2.2 release

[liyinan926]

Tasks:

• Update POMs to 2.2.0-k8s-0.5.0.
• Publish a new release v2.2.0-kubernetes-0.5.0.
• Build and publish docker images tagged v2.2.0-kubernetes-0.5.0.
• Update docs to refer to the new release version.

[foxish]

This is necessary for k8s 1.8 support.
cc @apache-spark-on-k8s/contributors

[liyinan926]

I got the following error when building driver-r image:

Step 3/6 : ADD R /opt/spark/R
ADD failed: stat /var/lib/docker/tmp/docker-builder221272199/R: no such file or directory

I do have R installed at /usr/local/bin/R. @ifilonenko any thoughts?

[liyinan926]

New release published https://github.com/apache-spark-on-k8s/spark/releases/tag/v2.2.0-kubernetes-0.5.0.

[liyinan926]

Commands used to build the distributions:

./dev/make-distribution.sh --pip --tgz --r -Pmesos -Pyarn -Pkinesis-asl -Phive -Phive-thriftserver -Pkubernetes -Phadoop-2.7 -Dhadoop.version=2.7.3
./dev/make-distribution.sh --pip --tgz -Pmesos -Pyarn -Pkinesis-asl -Phive -Phive-thriftserver -Pkubernetes -Phadoop-2.7 -Dhadoop.version=2.7.3 -Phadoop-provided
./sbin/build-push-docker-images.sh -r docker.io/kubespark -t v2.2.0-kubernetes-0.5.0 build
./sbin/build-push-docker-images.sh -r docker.io/kubespark -t v2.2.0-kubernetes-0.5.0 push

[liyinan926]

I think the problem is the following line in the Dockerfile:

ADD R /opt/spark/R

This will try to find R inside the unzipped distribution, which doesn't exist. @ifilonenko any thoughts on how to fix this?

[ifilonenko]

./dev/make-distribution.sh --r --pip --tgz -Pmesos -Pyarn -Pkinesis-asl -Phive -Phive-thriftserver -Pkubernetes -Phadoop-2.7 -Dhadoop.version=2.7.3

requires necessary R flag so script can run bash install-dev.sh for R packaging

[liyinan926]

Thanks @ifilonenko!

[liyinan926]

I tired adding the --r flag to make-distribution.sh and now I am getting the following error:

Error in loadVignetteBuilder(pkgdir, TRUE) :
  vignette builder 'knitr' not found

@ifilonenko @felixcheung any thoughts?

[liyinan926]

It worked after running R -e "install.packages(c('knitr', 'rmarkdown', 'testthat', 'e1071', 'survival'), repos='http://cran.us.r-project.org')".

[felixcheung]

cool!
for reference, please see https://github.com/felixcheung/spark-build/blob/master/Dockerfile
install.packages(c('curl', 'xml2', 'httr', 'devtools', 'testthat', 'knitr', 'rmarkdown', 'roxygen2', 'e1071', 'survival'), repos='http://cran.us.r-project.org/')

do you have roxygen2?

[liyinan926]

Seems --r cannot be used with -Phadoop-provided. I got the following error when doing so. Is this expected?

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream
	at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:127)
	at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:127)
	at scala.Option.getOrElse(Option.scala:121)
	at org.apache.spark.deploy.SparkSubmitArguments.mergeDefaultSparkProperties(SparkSubmitArguments.scala:127)
	at org.apache.spark.deploy.SparkSubmitArguments.<init>(SparkSubmitArguments.scala:113)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:113)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FSDataInputStream
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 7 more
Quitting from lines 48-50 (sparkr-vignettes.Rmd)
Error: processing vignette 'sparkr-vignettes.Rmd' failed with diagnostics:

[felixcheung]

Yes you need Hadoop to run the build with R. Sorry I didn't noticed earlier