Replace existing identity provider test fixtures with ones that support aarch64

[mark-vieira]

The :x-pack:test:idp-fixture project relies on a couple of Docker images that only support amd64 architecture, and therefore cannot be run on ARM. Specifically the unicon/shibboleth-idp and c2id/c2id-server images. We should look into finding alternatives here that support aarch64 so we aren't losing coverage on that platform.

[elasticmachine]

Pinging @elastic/es-delivery (Team:Delivery)

[elasticmachine]

Pinging @elastic/es-security (Team:Security)

[jakelandis]

I reached out to unicorn about updating their shibboleth-idp image and pretty confident they will not be updating their docker image. The good news is that the docker configuration is open source and there are a few forks, so we don't need to start from scratch to build an arm64 compatible image. The bad news is none of the images work out of the box so additional effort is needed for us to create an image, support arm64, and then update to the latest version.

Some issues:

The shibboleth nexus repo seems to have issue, but this link seems to have similar (same ?) downloads
https://build.shibboleth.net/maven/releases

https://shibboleth.net/downloads/identity-provider/$idp_version appears to be missing, but it is availble at https://shibboleth.net/downloads/identity-provider/archive/archive/$idp_version

http://central.maven.org is no more and replaced with https://repo.maven.apache.org

Simply bumping up jetty versions does not seem to work, but https://github.com/winstonhong/Shibboleth-SAML-IdP-and-SP/blob/master/shibboleth-idp-dockerized/Dockerfile is an example of a newer jetty version

Simply bumping the idp version in the Dockerfile does not seem to work but https://github.com/CSCfi/shibboleth-idp-dockerized/blob/master/Dockerfile has an example of a newer (4.x) idp

https://github.com/winstonhong/Shibboleth-SAML-IdP-and-SP/blob/master/shibboleth-idp-dockerized/ is the closest to working out of the box, but our tests (./gradlew :x-pack:qa:saml-idp-tests:javaRestTest --info) fail during the SSL handshake.

We can build from source located here : git clone https://git.shibboleth.net/git/java-identity-provider.git with Java11 and newer version of maven. However, we shouldn't need to due to https://shibboleth.net/downloads/identity-provider/archive/archive/$idp_version

The main blocker to getting the unicorn image to build is

DOWNLOAD: https://raw.githubusercontent.com/eclipse/jetty.project/master/jetty-server/src/test/config/etc/keystore?id=master to ${jetty.base}/etc/keystore
java.io.IOException: URL GET Failure [404/Not Found] on https://raw.githubusercontent.com/eclipse/jetty.project/master/jetty-server/src/test/config/etc/keystore?id=master
	at org.eclipse.jetty.start.fileinits.UriFileInitializer.download(UriFileInitializer.java:79)
	at org.eclipse.jetty.start.fileinits.UriFileInitializer.init(UriFileInitializer.java:60)
	at org.eclipse.jetty.start.BaseBuilder.processFileResource(BaseBuilder.java:283)
	at org.eclipse.jetty.start.BaseBuilder.processFileResources(BaseBuilder.java:375)
	at org.eclipse.jetty.start.BaseBuilder.build(BaseBuilder.java:239)
	at org.eclipse.jetty.start.Main.start(Main.java:406)
	at org.eclipse.jetty.start.Main.main(Main.java:76)

which is logged here: jetty/jetty.project#2138
however, I don't see any easy work around that since that is referenced (i think ) here, and not sure how to configure jetty 9.3.27 (especially via a docker image) to use a specific key store and it appears that jetty 9.4 has more changes that make this a bit more complex. I believe the keystore can be found here and dowloaded from https://github.com/eclipse/jetty.project/raw/jetty-9.3.27.v20190418/jetty-server/src/test/config/etc/keystore but not sure how to get the image to use that path.

[mark-vieira]

The bad news is none of the images work out of the box so additional effort is needed for us to create an image, support arm64

Realistically the only difference between x86 and arm images should be the included JRE. Everything else should be identical. So typically you'd be starting from a base JDK image, which if they are multi-arch, should actually just work transparently. So when building the fixture on arm it'd use the arm base image and x86 for intel.

It might be worth pointing out here that he issue with this is less about missing test coverage on ARM and more about local development. Essentially as it is, you cannot run these tests on an M1 Mac, which is what all Elastic developers will eventually be using and many already are. This means troubleshooting or developing these tests requires a) keeping intel-based hardware on hand, b) spinning up a vm or c) relying on the commit/push/pr-check feedback loop which is quite slow.

[jakelandis]

Success ! kinda ...

I forked the Unicorn repo https://github.com/jakelandis/shibboleth-idp-dockerized, and figured out the missing keystore config/download jakelandis/shibboleth-idp-dockerized@7f8b92b...master I need to clean up the image build (I was throwing stuff at the wall to see what stuck) support arm64, and eventually update the versions...but that all seems reasonable now.

The tests pass on my x86 box with the custom built docker image.

[mark-vieira]

#71378 (comment)

FWIW, we shoudln't need to go so far as to build/deploy these images anywhere. We can just stick the Dockerfile alongside the test fixture and configure docker compose to build it rather than pull an image. It's perhaps slightly less efficient but it's certainly simpler.

Again, for amd64 support it should be jsut a matter of ensure we using a multi-arch base image. In this case in the "build just in time" scenario I describe above, on ARM it should "just work" as when you build the Docker imge locally, it'll use the correct base image architecture automatically.