Cut a first release and announce

[ash211]

• Create branch-2.1-kubernetes based off branch-2.1 and rebase our code onto it (Prep for alpha release #177)
• Update README Update README #35
• Link to this repo from SPARK-18278
• Send our intention to cut a first release soon to spark-dev list
• Determine versioning scheme (v2.1.0-kubernetes-0.1.0-rc1 ?)
• Make go/no-go decisions on issues in the MVP umbrella Static Allocation MVP umbrella issue #9
• Tag release
• Build release tarballs and upload somewhere
• Add links to those release artifacts from README / github releases
• Link to this repo from official docs Link to this repo from official docs #34
• Send release announcement to the spark-users list

[ssuchter]

A question about the item "Build release tarballs and upload somewhere"...

We've started to think about this, and something occurred to us in that thought process. What version(s) of Hadoop will we produce artifacts for? If you look at http://spark.apache.org/downloads.html you'll notice in item 2. Select a package type there are packages for multiple different release branches of Hadoop, since Hadoop code is called from Spark at various places.

I think this might be appropriate for this Wednesday's discussion, but I wanted to raise the issue beforehand in case people wanted to start thinking about it.

[tnachen]

I don't think we need to decide around Hadoop versions as we can follow what latest release of Spark did

[ash211]

I'm most interested in the -hadoop2.7 and -without-hadoop versions but am happy to match Spark's version list, assuming no issues with compiling against older versions of Hadoop. Since we're not touching Hadoop much this should be no problem.

[ssuchter]

For the mechanical operation of publishing built artifacts, we recently came across functionality to get Travis to push builds from specific tagged commits as release artifacts to github.

http://stackoverflow.com/questions/12343452/how-to-publish-artifacts-in-travis-ci

I'll test this out in the new few days in my own repo and if it works will submit a PR (.travis.yml needs to be modified) for it. Then we will be able to just tag commits that we want our released artifacts to be built off of and we should get a pushed artifact.

[foxish]

Created labels 'in-alpha' and 'in-beta'. I think it would help to classify issues now, so that we know what we're fixing for alpha and what comes later, in order to prioritize.

[ash211]

Branch and tag creation process

Created a new branch called branch-2.1-kubernetes-release based off commit f9f5af4. That's the first commit where we had the branch-2.1-kubernetes set up properly.

Code freeze was on March 3rd, and the last commit to land on the old k8s-support-alternate-incremental pre-freeze was 4e72392.

Since then, these two commits have landed on the k8s-support-alternate-incremental branch:

• f27885a to propagate exceptions
• c118f47 for a minor logging change

Then we cherry picked all those commits over to the new branch-2.1-kubernetes and had a number of additional commits to get the build set up properly:

• 35724a3 mccheah 4 days ago Allow providing an OAuth token for authenticating against k8s
• fece639 Anirudh Ramanathan 9 days ago Adding clarification pre-alpha
• 8756494 Andrew Ash 9 days ago Add Apache license to a few files
• 8c08189 Anirudh Ramanathan 9 days ago Docs improvements
• b139b46 Andrew Ash 2 days ago Fix lint-check failures and javadoc8 break
• 2382ea6 Kimoon Kim 2 days ago Exclude flaky ExternalShuffleServiceSuite from Travis
• 3636939 Anirudh Ramanathan 4 days ago Fix pom versions

Since that prep-for-alpha-release branch merged into branch-2.1-kubernetes, there's been one more commit:

• d5502ed Andrew Ash 5 minutes ago Allow the driver pod's credentials to be shipped from the submission client (squashed)

Building the release branch off the commit right before that (the first ready commit of branch-2.1-kubernetes) gives us f9f5af4 as a starting point.

Then I created an empty commit on that new branch for GH to place the v2.1.0-kubernetes-0.1.0-rc1 tag on, so that git describe on the main branch-2.1-kubernetes doesn't pick up that tag. The new branch is also protected in GH to prevent accidental pushes or evading the CI checks.

Then I created the v2.1.0-kubernetes-0.1.0-rc1 tag on branch-2.1-kubernetes-release using the GH release UI.

Next steps in the release process

At this point now, we should:

1. create release tarballs and link them in the https://github.com/apache-spark-on-k8s/spark/releases/tag/v2.1.0-kubernetes-0.1.0-rc1 release notes
2. create docker images and link them in the release notes

@foxish since you've done the second part earlier, would you be able to create new docker images off this new RC1 release tag?

@ssuchter you mentioned a couple comments above that Travis could possibly push release artifacts to GitHub. Now that we've moved to Jenkins, does it make sense to have a release tag created in GH kick off the release process from there? If it could build the several dists (various versions of Hadoop) and also build and publish the docker images, we could have a one-click release process through the GH UI which would be pretty awesome. Do you have bandwidth to start investigating automating the release process in a future release? I'm guessing we want to get this first RC out there earlier though so don't think we should block RC1 on the Jenkins setup.

[ssuchter]

I agree we shouldn’t block the initial release with the Jenkins/GH tag integration, but the scheme you describe seems like a good one to me. I or someone else here will look into it.

[erikerlandson]

With respect to docker images, I created a docker org 'k8s4spark' where they could live, corresponding to this 'apache-spark-on-k8s' GH org.

[ash211]

@erikerlandson / @foxish I think you guys have both created images on docker -- can you please deconflict off-ticket and reply back here with what org to be publishing to?

[foxish]

Discussed offline. Going with kubespark. I'll push new images from RC1 and send another docs PR updating tags.

[ash211]

Effectively done now, though we didn't announce the first release on the mailing list. I propose we wait until the second version is released pre-Spark Summit and send a release announcement for that version instead.

[foxish]

+1