make build-and-push-images fails on M1

[henrjk]

For me make build-and-push-images fails. Below is an excerpt

make build-and-push-images 
cd scripts && ./build-and-push-images.sh
Building image localhost:5000/ods/ods-buildah...
[+] Building 1.6s (18/18) FINISHED                                                                                                                                                                                                        
...
Building image localhost:5000/ods/ods-finish...
...
Building image localhost:5000/ods/ods-go-toolset...
...
Building image localhost:5000/ods/ods-gradle-toolset...
...
Building image localhost:5000/ods/ods-helm...
[+] Building 9.6s (15/21)                                                                                                                                                                                                                 
 => [internal] load build definition from Dockerfile.helm                                                                                                                                                                            0.0s
 => => transferring dockerfile: 2.18kB                                                                                                                                                                                               0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                    0.0s
 => => transferring context: 2B                                                                                                                                                                                                      0.0s
 => [internal] load metadata for registry.access.redhat.com/ubi8/ubi-minimal:8.4                                                                                                                                                     0.5s
 => [internal] load metadata for registry.access.redhat.com/ubi8/go-toolset:1.16.12                                                                                                                                                  0.5s
 => [builder 1/9] FROM registry.access.redhat.com/ubi8/go-toolset:1.16.12@sha256:287d341349331a19866a2862a55e0cc5c412f21a86ba80629f59579ef6a38c03                                                                                    0.0s
 => [internal] load build context                                                                                                                                                                                                    0.0s
 => => transferring context: 419.82kB                                                                                                                                                                                                0.0s
 => [stage-1 1/7] FROM registry.access.redhat.com/ubi8/ubi-minimal:8.4@sha256:c536d4c63253318fdfc1db499f8f4bb0881db7fbd6f3d1554b4d54c812f85cc7                                                                                       0.0s
 => CACHED [stage-1 2/7] RUN microdnf install skopeo-1.4* git-2.27* tar-1.30* && microdnf clean all                                                                                                                                  0.0s
 => CACHED [builder 2/9] RUN mkdir -p /tmp/helm     && cd /tmp     && curl -LO https://get.helm.sh/helm-v3.5.2-linux-amd64.tar.gz     && tar -zxvf helm-v3.5.2-linux-amd64.tar.gz -C /tmp/helm     && mv /tmp/helm/linux-amd64/helm  0.0s
 => CACHED [builder 3/9] COPY go.* .                                                                                                                                                                                                 0.0s
 => CACHED [builder 4/9] COPY cmd cmd                                                                                                                                                                                                0.0s
 => CACHED [builder 5/9] COPY internal internal                                                                                                                                                                                      0.0s
 => CACHED [builder 6/9] COPY pkg pkg                                                                                                                                                                                                0.0s
 => CACHED [builder 7/9] RUN cd cmd/deploy-with-helm && CGO_ENABLED=0 go build -o /usr/local/bin/deploy-with-helm                                                                                                                    0.0s
 => ERROR [builder 8/9] RUN mkdir -p /tmp/sops     && cd /tmp/sops     && curl -LO https://github.com/mozilla/sops/releases/download/v3.7.1/sops-3.7.1-1.x86_64.rpm     && yum install -y sops-3.7.1-1.x86_64.rpm     && sops --ver  9.0s
------                                                                                                                                                                                                                                    
 > [builder 8/9] RUN mkdir -p /tmp/sops     && cd /tmp/sops     && curl -LO https://github.com/mozilla/sops/releases/download/v3.7.1/sops-3.7.1-1.x86_64.rpm     && yum install -y sops-3.7.1-1.x86_64.rpm     && sops --version:         
#15 0.235   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current                                                                                                                                                 
#15 0.235                                  Dload  Upload   Total   Spent    Left  Speed                                                                                                                                                   
100   660  100   660    0     0   2608      0 --:--:-- --:--:-- --:--:--  2608                                                                                                                                                            
100 12.7M  100 12.7M    0     0  6871k      0  0:00:01  0:00:01 --:--:-- 8347k                                                                                                                                                            
#15 2.383 Updating Subscription Management repositories.
#15 2.383 Unable to read consumer identity
#15 2.387 
#15 2.387 This system is not registered with an entitlement server. You can use subscription-manager to register.
#15 2.387 
#15 4.460 Red Hat Universal Base Image 8 (RPMs) - BaseOS  283 kB/s | 570 kB     00:02    
#15 6.616 Red Hat Universal Base Image 8 (RPMs) - AppStre 1.1 MB/s | 2.3 MB     00:02    
#15 8.737 Red Hat Universal Base Image 8 (RPMs) - CodeRea 6.9 kB/s |  13 kB     00:01    
#15 8.897 Error: 
#15 8.897  Problem: conflicting requests
#15 8.897   - package sops-3.7.1-1.x86_64 does not have a compatible architecture
#15 8.898 (try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)
------
executor failed running [/bin/bash -o pipefail -c mkdir -p /tmp/sops     && cd /tmp/sops     && curl -LO https://github.com/mozilla/sops/releases/download/v${SOPS_VERSION}/sops-${SOPS_VERSION}-1.x86_64.rpm     && yum install -y sops-${SOPS_VERSION}-1.x86_64.rpm     && sops --version]: exit code: 1
make: *** [build-and-push-images] Error 1                                           

The above is from the second run. The full log from the first run is at build-and-push-images.log

Here is some info about the machine

uname -a
Darwin <host-name-redacted> 21.2.0 Darwin Kernel Version 21.2.0: Sun Nov 28 20:28:41 PST 2021; root:xnu-8019.61.5~1/RELEASE_ARM64_T6000 arm64

 docker version
Client:
 Cloud integration: v1.0.22
 Version:           20.10.11
 API version:       1.41
 Go version:        go1.16.10
 Git commit:        dea9396
 Built:             Thu Nov 18 00:36:09 2021
 OS/Arch:           darwin/arm64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.11
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.9
  Git commit:       847da18
  Built:            Thu Nov 18 00:34:44 2021
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.4.12
  GitCommit:        7b11cfaabd73bb80907dd23182b9347b4245eb5d
 runc:
  Version:          1.0.2
  GitCommit:        v1.0.2-0-g52b36a2
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

I believe docker build builds for the os platform by default (arm64) and succeeds for ods-buildah, ods-finish, ods-go-toolset, ods-gradle-toolset but fails for ods-helm

[henrjk]

I relaxed locally build-and-push-images.sh so it continues with the next image if one fails. The other build that fails is for the ods-node16-typescript-toolset.

See build-and-push-ts-image.log for the full log.

[henrjk]

Until finding a fix so that images run natively on M1 added fix so that helm and typescript images are build for platform linux/amd64 (see commit c45c2e6) for details.

Building for linux/amd64 on M1 is much slower than directly on a Intel based Mac:

• helm: M1 456.6s vs Intel 95.1s
• typescript: M1 469.6s vs Intel 120.6s

However after a make install-ods-tasks-kind to deploy the tasks make test-tasks now the TestTaskODSDeployHelm PASSes. All other tests fail as they require bitbucket, sonarqube or nexus. See related issues linked above.

[michaelsauter]

Thanks for experimenting with this and reporting all of it. That doesn't look too promising :(

Let me try to summarise:

1. our own images work except Helm and Typescript
2. external images do not work except Postgres

Correct?

I guess for (1) we could try to adapt the Dockerfile to make them build also for arm64. I think this should be doable. For example, we can install sops via go install go.mozilla.org/sops/v3/cmd/sops@v3.7.1 and Helm offers a arm64 release. The Dockerfile could automatically select the correct binary based on a build arg maybe? We could use ARGs aligned with https://docs.docker.com/engine/reference/builder/#automatic-platform-args-in-the-global-scope, even if we do not use buildx (locally we could but we also need to ensure it still works within the BuildConfig context).

For (2) we could open issues in the GitHub repos of the official images to see if support for arm64 is possible/planned? For example for Nexus, I found https://github.com/klo2k/nexus3-docker, which uses the official Dockerfile with a few alterations.

[henrjk]

@michaelsauter Your summary is correct.
I agree with your proposals but not sure yet whether a build-arg is needed -- need to educate myself first a bit more. But perhaps others have more multi platform docker experience already..

[michaelsauter]

Cool. I've made the small suggested change in #364.

I am aware of the following four ways to build for multiple architectures:

• Using methods like in Install SOPS via go install #364 that do not hardcode an architecture
• Using commands like uname to figure out the architecture dynamically (I don't particularly like it because it can be brittle and will duplicate logic between Dockerfiles)
• Using a build arg as suggested
• Using a different Dockerfile per architecture (I don't particularly like it because it increases maintenance effort)

[henrjk]

https://github.com/docker-library/official-images#multiple-architectures has some suggestions regarding conditional in the RUN commands to target various userspace architectures, to quote dpkg --print-architecture or apk -print-arch to detect the userspace architecture. Only use uname for architecture detection when more accurate tools cannot be installed.
It also links to https://github.com/docker-library/golang/blob/b879b60a7d94128c8fb5aea763cf31772495511d/1.16/buster/Dockerfile#L24-L68

[henrjk]

Also this could sometimes be of relevant: https://github.com/sixeyed/diamol/tree/master/ch16/lab

[henrjk]

Article Faster Multi-Platform Builds: Dockerfile Cross-Compilation Guide - Docker Blog has a section about predefined variables such as BUILDPLATFORM and TARGETPLATFORM.

[henrjk]

Tried out a fix for ods-deploy as shown in commit 1638d40 but getting this on the M1 mac:

→ ./build-and-push-images.sh --image helm    
Building image localhost:5000/ods/ods-helm...
[+] Building 2.5s (21/21) FINISHED                                                                                                                                                                                    
 => [internal] load build definition from Dockerfile.helm                                                                                                                                                        0.0s
 => => transferring dockerfile: 2.06kB                                                                                                                                                                           0.0s
 => [internal] load .dockerignore                                                                                                                                                                                0.0s
 => => transferring context: 2B                                                                                                                                                                                  0.0s
 => [internal] load metadata for registry.access.redhat.com/ubi8/ubi-minimal:8.4                                                                                                                                 0.6s
 => [internal] load metadata for registry.access.redhat.com/ubi8/go-toolset:1.16.12                                                                                                                              0.6s
 => [builder 1/9] FROM registry.access.redhat.com/ubi8/go-toolset:1.16.12@sha256:287d341349331a19866a2862a55e0cc5c412f21a86ba80629f59579ef6a38c03                                                                0.0s
 => [stage-1 1/7] FROM registry.access.redhat.com/ubi8/ubi-minimal:8.4@sha256:c536d4c63253318fdfc1db499f8f4bb0881db7fbd6f3d1554b4d54c812f85cc7                                                                   0.0s
 => [internal] load build context                                                                                                                                                                                0.0s
 => => transferring context: 5.93kB                                                                                                                                                                              0.0s
 => CACHED [stage-1 2/7] RUN microdnf install skopeo-1.4* git-2.27* tar-1.30* && microdnf clean all                                                                                                              0.0s
 => CACHED [builder 2/9] RUN mkdir -p /tmp/helm     && cd /tmp     && curl -LO https://get.helm.sh/helm-v3.5.2-linux-arm64.tar.gz     && tar -zxvf helm-v3.5.2-linux-arm64.tar.gz -C /tmp/helm     && mv /tmp/h  0.0s
 => CACHED [builder 3/9] COPY go.* .                                                                                                                                                                             0.0s
 => CACHED [builder 4/9] COPY cmd cmd                                                                                                                                                                            0.0s
 => CACHED [builder 5/9] COPY internal internal                                                                                                                                                                  0.0s
 => CACHED [builder 6/9] COPY pkg pkg                                                                                                                                                                            0.0s
 => CACHED [builder 7/9] RUN cd cmd/deploy-with-helm && CGO_ENABLED=0 go build -o /usr/local/bin/deploy-with-helm                                                                                                0.0s
 => CACHED [builder 8/9] RUN go install go.mozilla.org/sops/v3/cmd/sops@v3.7.1     && sops --version                                                                                                             0.0s
 => CACHED [builder 9/9] RUN go install filippo.io/age/cmd/...@v1.0.0     && age --version                                                                                                                       0.0s
 => CACHED [stage-1 3/7] COPY --from=builder /usr/local/bin/deploy-with-helm /usr/local/bin/deploy-with-helm                                                                                                     0.0s
 => CACHED [stage-1 4/7] COPY --from=builder /usr/local/bin/helm /usr/local/bin/helm                                                                                                                             0.0s
 => CACHED [stage-1 5/7] COPY --from=builder /usr/local/bin/sops /usr/local/bin/sops                                                                                                                             0.0s
 => CACHED [stage-1 6/7] COPY --from=builder /usr/local/bin/age /usr/local/bin/age                                                                                                                               0.0s
 => ERROR [stage-1 7/7] RUN mkdir -p $HELM_PLUGINS     && HELM_DATA_HOME=${HELM_PLUGINS%/*} helm plugin install https://github.com/databus23/helm-diff --version v${HELM_PLUGIN_DIFF_VERSION}     && HELM_DATA_  1.8s
------
 > [stage-1 7/7] RUN mkdir -p $HELM_PLUGINS     && HELM_DATA_HOME=${HELM_PLUGINS%/*} helm plugin install https://github.com/databus23/helm-diff --version v${HELM_PLUGIN_DIFF_VERSION}     && HELM_DATA_HOME=${HELM_PLUGINS%/*} helm plugin install https://github.com/jkroepke/helm-secrets --version v${HELM_PLUGIN_SECRETS_VERSION}     && ls -lah $HELM_PLUGINS     && sops --version     && age --version:
#21 1.722 No prebuild binary for linux-arm64.
#21 1.722 Failed to install helm-diff
#21 1.722 	For support, go to https://github.com/databus23/helm-diff.
#21 1.722 Error: plugin install hook for "diff" exited with error
------
executor failed running [/bin/sh -c mkdir -p $HELM_PLUGINS     && HELM_DATA_HOME=${HELM_PLUGINS%/*} helm plugin install https://github.com/databus23/helm-diff --version v${HELM_PLUGIN_DIFF_VERSION}     && HELM_DATA_HOME=${HELM_PLUGINS%/*} helm plugin install https://github.com/jkroepke/helm-secrets --version v${HELM_PLUGIN_SECRETS_VERSION}     && ls -lah $HELM_PLUGINS     && sops --version     && age --version]: exit code: 1
   

[henrjk]

For helm-diff there is already a PR at databus23/helm-diff#267