OpenJDK 8u212 and 11.0.3 image quality concerns

[vielmetti]

I'm reading

https://mail.openjdk.java.net/pipermail/jdk8u-dev/2019-May/009330.html

entitled "Mystery meat OpenJDK builds strike again", in which it is noted that the openjdk:8 and openjdk:11 versions were observed to be not release versions but instead pre-release versions, originating from Debian and not reflecting the actual final releases of those versions.

[tianon]

@robilad we could definitely use some "official" advice here if you could spare any or refer us to any other authoritative folks willing to help out 😅

If these builds are not representing OpenJDK well, I'd like to remove or more preferably replace them, but I want to make sure that we don't just trade one sideways situation for another.

To be totally clear, this sort of oddness with the Debian builds is exactly why for 12+ we've switched to be based entirely on the builds officially recommended by the OpenJDK project (https://jdk.java.net/, #235). There's an ongoing discussion in #302 to add back Debian-based images, but they'd still be based on those "official" builds (not Debian's packages).

I also think it's reasonable to point out docker-library/official-images#5710 here, which is an initiative from the AdoptOpenJDK project to add official AdoptOpenJDK images maintained directly by that community. 👍

Perhaps if that were available as a viable alternative for users who need these older versions, we could more seriously consider simply dropping any builds older than OpenJDK 12 (given that 12+ is the only place we've got officially-recommended binaries from OpenJDK officially)?

[vielmetti]

attn @giltene

[tianon]

I think it also makes sense to point out https://hub.docker.com/_/amazoncorretto and https://hub.docker.com/_/ibmjava as viable official-images alternatives maintained by their respective upstreams, although with the caveat that the latter is likely to be superseded by AdoptOpenJDK.

[giltene]

@tianon I'd be happy to provide constructive advice and direction on how the Official Docker images for OpenJDK might be able to regain credibility over time.

The first step in doing that is to avoid "making excuses for" the severity of the current situation (and the identical situation that occurred in 2014-2015, referred to here: 00a9c5c#commitcomment-8032128 ). It happened. That sucks. Now let's work on fixing things so it doesn't happen again.

By now, it should be all too apparent that the debian builds that have been used by the official docker images for OpenJDK to satisfy your users' needs for a current, updated OpenJDK are/were the wrong tool for the job. Stable debian builds are often dramatically out of date (e.g. the current stable builds are at least 9 months behind on bug fixes and security updates), and the seemingly-more-up-to-date builds found in various non-default debian repos (unstable, stretch-backports, etc.) are regularly mislabeled in a way that makes it "hard" to differentiate bad (unreleased, built from top-of-tree, and full of bugs and security holes not expected in the version that they report being) from good builds.

Since using debian builds has caused this serious issue across multiple versions and significant time periods (at the very least for 8u40, 8u212, and 11.0.3, but may have happened multiple other times in between for all I know), repeating that mistake a third time is the obvious thing we need to avoid if we intend to rebuild confidence in the trustworthiness of official docker images for OpenJDK. The only reasonable remedy is (obviously) to stop using debian builds in official docker images for openjdk, and to find an alternative, reputable and reliable source for JDK and JRE builds for use in your official images.

This was suggested before (see link above), but not acted on at the time. I believe this was driven by misconceptions that time, distance, and the severity of the current situation have hopefully helped clarify by now. At the core (and seemingly the reason you did not switch to Zulu back then and prevent the problem from happening again) was the reaction to the proposed solution having a "brand", and the notion that if something's "name" is spelled differently than "openjdk" it is not a build of openjdk, but a "compatible variant" that is somehow suspect. In fact (and with 4+ year track record since then), the opposite is demonstrably true. With the appearance of various OpenJDK builds over the years, you will note that ALL reputable builds of OpenJDK that are distributed outside of an OS's native repo have "name" brand labels, for the simple reason that the OpenJDK trademark fair use rules demand it (and reputable sources tend to try to adhere to trademark rules). They might say things like "Zulu is a build of OpenJDK", but will carefully avoid using "OpenJDK" in an actual name. The "brand name" of the thing you will want to download may be "Zulu", or "IcedTea", or "Corretto", or "Liberica", or "AdoptOpenJDK", but it will never be plain "OpenJDK".

If the OS distro that you base your images on happens to distribute high quality, timely updates of OpenJDK on a regular and reliable basis, you may be ok using the build-in repos to get the JDKs for these images. But if not, and you end up needing a reliable source for JDKs and JREs to include in your Official Docker images for OpenJDK for some OSs (as you clearly needed to do to get openjdk:8 up to date), you need an alternative and reliable source for JDK and JRE binaries.

I would obviously recommend Zulu's 100% free, 100% OSS, reliably delivered builds for those OSs for this purpose. By now, the fit and quality of Zulu for this purpose should be well-established. Zulu's long track record makes it clear where good 100% free, 100% OSS, never-mislabeled, always tested builds of OpenJDK can be reliably found on the internet.

Recently, some other potential sources for OpenJDK builds have appeared, including several newcomers with good potential for establishing a reputation by showing consistent delivery. At this point in time, however, when it comes to both breadth (in platforms, packages, availability via public repos, etc.) and length of actual demonstrated delivery (number of good quality quarterly builds delivered for free, with an established track record of NOT delivering bad material), Zulu is unmatched as a reliable and reliably free source of OpenJDK builds.

Since Zulu is available (amongst many other delivery forms) through a deb repo, the following example shows the straightforward changes that would allow you to cleanly transition your current openjdk:8 Dockerfile to one that delivers a good (actually released) 8u212:

18a19,20
> RUN apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 0xB1998361219BD9C9
> RUN echo "deb http://repos.azulsystems.com/debian stable  main" >> /etc/apt/sources.list.d/zulu.list
39c41
< RUN ln -svT "/usr/lib/jvm/java-8-openjdk-$(dpkg --print-architecture)" /docker-java-home
---
> RUN ln -svT "/usr/lib/jvm/zulu-8-$(dpkg --print-architecture)" /docker-java-home
43c45
< ENV JAVA_DEBIAN_VERSION 8u212-b01-1~deb9u1
---
> ENV JAVA_ZULU_VERSION 8.38.0.13
54c56
< 		openjdk-8-jdk="$JAVA_DEBIAN_VERSION" \
---
> 		zulu-8="$JAVA_ZULU_VERSION" \

[vielmetti]

Noting geonetwork/docker-geonetwork#23 as related.

[adinn]

If you are looking for Linux binary releases for the jdk8u and jdk11u projects you can find both: latest official releases and latest early access versions for the next upcoming release at the following URL:

https://adoptopenjdk.net/upstream.html

As the blurb says:

"These binaries are built by Red Hat on behalf of the OpenJDK jdk8u and jdk11u projects. The binaries are created from the unmodified source code at OpenJDK. Support is not provided."

i.e. these are vanilla binaries that 1) 100% reflect the shared, public repo code and 2) should work on all current, mainstream Linux releases (Debian included).

Note: that last sentence about 'support' simply means you don't have a phone to ring or an email to post to if you want help using them. If you want either of those you need to pay someone like Red Hat, Azul, AdoptOpenJDK etc.

Note: It does not mean these binaries are a one-off thrown over the wall release. The EA binaries will be updated regularly as new fixes are added to the jdk8u and jdk11u projects, The official binaries will be updated with security patches and critical fixes every 3 months when the security embargo is lifted.

Note: although it is not relevant in this forum there are also Windows releases at the above URL provided under the same terms.

[theRealAph]

"These binaries are built by Red Hat on behalf of the OpenJDK jdk8u and jdk11u projects. The binaries are created from the unmodified source code at OpenJDK. Support is not provided."

i.e. these are vanilla binaries that 1) 100% reflect the shared, public repo code and 2) should work on all current, mainstream Linux releases (Debian included).

Note: that last sentence about 'support' simply means you don't have a phone to ring or an email to post to if you want help using them. If you want either of those you need to pay someone like Red Hat, Azul, AdoptOpenJDK etc.

Note: It does not mean these binaries are a one-off thrown over the wall release. The EA binaries will be updated regularly as new fixes are added to the jdk8u and jdk11u projects, The official binaries will be updated with security patches and critical fixes every 3 months when the security embargo is lifted.

Indeed. I should do something about getting that wording fixed.

[tianon]

It seems as though there's been some misunderstanding, so I want to start with a bit of a level set.

1. Add initial all-Oracle variants (oraclelinux + jdk.java.net tarballs) #235 was done in collaboration with @robilad as an upstream representative to add new variants which consume the official OpenJDK artifacts from https://jdk.java.net/ which are linked from https://openjdk.java.net/ (specifically because we knew the Debian package situation was not great and we needed to do something)
2. they were new variants to avoid backwards compatibility concerns for existing image users
3. the fallout from our users on that action is in Where's the OpenJDK 12 image? #302, where we're not discussing adding back Debian's packages, but are instead discussing taking those same official https://jdk.java.net/ artifacts and putting them in a Debian base in addition to what we have
4. for OpenJDK 12 and 13, we only have images which are consuming from https://jdk.java.net/, and have no plans to expand beyond that without an official upstream recommendation (preferably via an update to https://openjdk.java.net/ and/or https://jdk.java.net/)
5. https://twitter.com/tianon/status/1118670830626410496

   to be 110% clear, we would love and welcome the OpenJDK project to officially maintain the image instead of us, and would happily help coordinate that

Regarding Azul-based images, I think an Azul Zulu official image would be a great addition to the program, and I'd suggest taking a look at https://github.com/docker-library/official-images#contributing-to-the-standard-library -- I'm also happy to help do the actual legwork to help make that happen (whether that's writing Dockerfiles, making the PR over there, just advising, etc).

While I agree that Azul's Java packages are likely better than Debian, I don't think they're any more appropriate as an "official" OpenJDK image than Debian's are, so if we're going to switch from one unofficial package to another, I'd rather consider dropping 8 and 11 and suggesting users look at some of the alternatives that already exist in the official images today.

Regarding https://adoptopenjdk.net/upstream.html, I personally think that's a fantastic initiative and hope that it gains more traction. However, we asked @robilad about that and were given a similar answer to the concerns above -- essentially that they aren't artifacts officially released from the OpenJDK project, and thus don't seem appropriate for use here (which is why we're discussing the problems with the Debian-released artifacts in the first place, so IMO it's fair). I think this is something to work out within the OpenJDK project directly (and again, preferably get https://openjdk.java.net/ and/or https://jdk.java.net/ updated to note the outcome -- even noting the outcome publicly in the case of "no, these won't be considered official by the project" would be useful for users looking to know more about these artifacts IMO).

So here's what I think our next steps are (specifically in the context of the Docker image maintained here, https://hub.docker.com/_/openjdk -- hopefully someone involved in the OpenJDK community can take further community concerns and champion them over there):

1. ensure Add AdoptOpenJDK Images (Includes both HotSpot and Eclipse OpenJ9) official-images#5710 continues to get the love and attention it needs to get an official AdoptOpenJDK image (for further Java-diversity in the official images)
2. possibly get a new Azul-based image we can recommend to users as well? 👍
3. update the image description on https://hub.docker.com/_/openjdk to both point out where the 12 and 13 image artifacts come from and to officially note the deprecation of the 8 and 11 artifacts, officially suggesting that users of those versions consider alternatives (which we'll link directly to) such as adoptopenjdk, amazoncoretto, ibmjava (and hopefully Azul Zulu?)

[adinn]

I'm not really sure what you (or, perhaps I mean Dalibor Topic of Oracle -- aka @robilad) mean by this comment:

"Regarding https://adoptopenjdk.net/upstream.html ... we asked @robilad about that and were given a similar answer to the concerns above -- essentially that they aren't artifacts officially released from the OpenJDK project, and thus don't seem appropriate for use here."

I think you have been taking advice from the wrong person. Dalibor has pointed you at Oracle-built binaries for jdk11u and jdk8u. They can in no way constitute an an 'official' release since Oracle are deliberately not participating in the OpenJDK updates project for jdk11u and jdk8u. Oracle explicitly devolved all responsibility for updates of those releases to non-Oracle members of the OpenJDK project. Oracle are continuing to provide their own proprietary versions of jdk11u and jdk8u based on their own private versions of these repositories but they are not co-operating with the project in maintaining the open source code base for jdk11u and jdk8u. Taking releases from them is like asking Microsoft to provide you with a Linux release.

By contrast, the releases at https://adoptopenjdk.net/upstream.html were built by Red Hat on behalf of the OpenJDK jdk11u and jdk8u update projects explicitly to provide 'official', upstream, vanilla releases independent from any releases provided by vendors. They have been built specially by Red Hat for this purpose as universal releases and,note well, are not the same as the releases Red Hat ships with its operating system releases even though they are based on the same sources.

Most importantly, I describe these releases as 'official' with a justification that Dalibor or Oracle cannot possibly pretend to. The jdk11u and jdk8u updates project lead (Andrew Haley, a Red Hat employee and OpenJDK governing board member at large i.e. elected representative of the OpenJDK community) explicitly asked for these releases to be produced and made available as updates projects releases. If Dalibor wants to use that word to describe the Oracle releases then I think you need to check which end he is talking out of.

[tianon]

Please keep the discussion civil and assume good faith.

As far as I am aware, https://openjdk.java.net/ is authoritative on OpenJDK -- is that not correct? Is there a better source we should be considering for authoritative information about OpenJDK, especially about the official status of https://adoptopenjdk.net/upstream.html?

[tianon]

Also, I want it to be clear that this was me paraphrasing @robilad's sentiment because I couldn't find the exact source -- he did not use those exact words, so attacking him is not only inappropriate, it is also unfair to him.

[adinn]

"As far as I am aware, https://openjdk.java.net/ is authoritative on OpenJDK -- is that not correct?"

No, not with regard to non-Oracle releases. Oracle will not let non-Oracle members of the OpenJDK project install releases on openjdk.java.net. So, the OpenJDK updates projects for jdk8u and jdk11u have been forced to distribute official binaries via another site. By contrast, Oracle have unilaterally placed their jdk11u and jdk8u releases on that site without any sanction from the updates project.

So, rather than ask those actually involved in the updates project whether their releases are 'official' you might perhaps want to ask Oracle (or at least the Oracle representative you talked to) why they consider it appropriate to be claiming that they are in a position to distribute official releases for an open source project they have quite explicitly refused to continue to be involved in.

"Is there a better source we should be considering for authoritative information about OpenJDK, especially about the official status of https://adoptopenjdk.net/upstream.html?"

You could perhaps take that up with Andrew Haley (aph@redhat.com) or, perhaps, Doug Lea (dl@cs.oswego.edu). Andrew is the current project lead for the jdk8u and jdk11u updates projects. He and Doug are the two people who were voted onto the OpenJDK governing board to represent the OpenJDK community.

I am sorry you considered my last response uncivil. I was trying very hard to be restrained in the face of what I (still) see as an extremely misguided, if not abusive, action. If I had posted what I wanted to say the air would have been much more blue.

Also, I should point out that this is not the first time that Dalibor has made public comments about the state of the updates project that have muddied the waters about the status of the various available releases, So I have a reason to be skeptical of his bona fides. I'd be very interested not only to hear Dalibor's exact words but also his explanation of why he thinks Oracle should be pushing out releases for the open project.

[theRealAph]

Please keep the discussion civil and assume good faith.

As far as I am aware, https://openjdk.java.net/ is authoritative on OpenJDK -- is that not correct? Is there a better source we should be considering for authoritative information about OpenJDK, especially about the official status of https://adoptopenjdk.net/upstream.html?

Yes, there is a better source: me. I am the OpenJDK 8 and 11 Updates OpenJDK project lead. If @robilad is saying that the builds at https://adoptopenjdk.net/upstream.html are not "official", I'd disagree. He is not involved in the upstream 8u and 11u projects.

While it is true that the OpenJDK Governing Board has not sanctioned those releases, they (or rather we, since I am a member) didn't sanction Oracle's OpenJDK releases either. As far as I am aware, the lead of an OpenJDK project is entitled to release binary builds, and there is clearly a need for them.

I'm happy to help with any further questions that you have.

[giltene]

@tianon I think you misunderstand the purpose of the reference implementation builds linked to in https://jdk.java.net/ . Those are certainly not the builds anyone will want to use if they expect updates after the first few months of a new Java version coming out. They stop being updated after the first 6 months.

The reference implementations pointed to there are useful for soon-after-initial-GA releases only. They are updated for 6 months after GA and then stopped. For the non-LTS "Feature releases" (e.g. 9, 10, 12, 13, ...), where the projects do not get further updates and move on to the next version, this may be fine. But for the long term releases (7, 8, 11, 17, ...), which are the versions that the vast majority of people will actually need to run, no updates will appear. Specifically: 11 LTS is not being updated there anymore (11.0.2 is the last one you'll see there, apparently), and the latest version of 8 linked to there is 8u40 (which is 4 years old).

The bottom line is that there is no "official place to get OpenJDK binaries" and there never has been, because OpenJDK is a source code project. Downstream builds are made and posted by various parties, in various places. Some are good, some not so good, and some are dangerous to use.

You have no choice but to choose a source you can trust. And (especially after the history that led us here), that trust should probably be based on a demonstrated history of delivering non-bad builds.

While some good (and blessed by the project lead) material can be found in other places, e.g. the recently created https://adoptopenjdk.net/upstream.html mentioned above, which covers a handful of setups (jdk only, tarball/zip only, glibc only for linux), I'll repeat my note that Zulu has been providing reliable builds of OpenJDK for several years now. The Zulu build matrix covers 100s of packages every quarter, and we test and certify each to make sure people can trust their provenance. These include a multitude of packages that are often needed by containers [jdks and jres, with and without openjfx, linux glibc, Alpine-native (musl libc), windows, mac, etc.], and delivery formats [tarballs, zips, dmgs, deb packages, rpm packages, msi packages, public deb and rpm repos, etc.].

You can keep looking around, and may end up picking some non-Zulu build that will satisfy your needs, but I urge you to make a choice and fix this quickly, as every additional day that passes by with bad (literally insecure) versions being delivered to users of the default official docker hub images for java 8 and 11 (e.g. what people get when they run "docker run -it --rm openjdk:8 java -version" right now) is causing lasting damage to the reputation of these images.

[theRealAph]

The bottom line is that there is no "official place to get OpenJDK binaries" and there never has been, because OpenJDK is a source code project. Downstream builds are made and posted by various parties, in various places. Some are good, some not so good, and some are dangerous to use.

I suppose we're quibbling about what the word "official" means, or can mean, in an open-source project. While the OpenJDK project itself doesn't bless binaries, I believe that it is in the interests of the OpenJDK Updates projects (and those of our users) to make reference binaries available. Whether these are "official" or not depends on your precise definition of the word: in its adjectival form the word means "Having the approval or authorization of an authority," and as the Project Lead I think I qualify. But in the end, as you say, it's up to the user to make a judgment.

Having said that, I agree with you on this:

I urge you to make a choice and fix this quickly, as every additional day that passes by with bad (literally insecure) versions being delivered to users of the default official docker hub images for java 8 and 11 (e.g. what people get when they run "docker run -it --rm openjdk:8 java -version" right now) is causing lasting damage to the reputation of these images.

Docker deserves a proper, compliance-tested OpenJDK image.