Automatically build docker images with CircleCI/GitLab CI

[saraedum]

It would be nice to update our docker images automatically through continuous integration services. Of course it's good to have these images up-to-date without manual intervention but this is also convenient as a starting point for people who want to use CI for their own branches of Sage (besides the patchbot.)¹

This ticket proposes recipes for GitLab CI and CircleCI to build our docker images automatically. On the respective websites, the CI can be configured to push automatically to the Docker Hub. A webhook (on github) updates the README on Docker Hub automatically.

I implemented this for both GitLab CI and CircleCI. I think GitLab CI is more relevant in the long run, also it's open source and people can provision their own machines as runners. CircleCI at the same time works out of the box for Sage without private test runners and it also allows for easier debugging as you can logon to the machine running your tests with SSH. I tried to share most code between the two implementations.

See also sagemath/docker-images#13 and sagemath/sage-binder-env#3 for a followup (automatically provide jupyter notebooks for easier review.)

---

Here are some numbers and screenshots (click on the screenshots to go to the real pages):

GitLab CI

If I provision my own runner from Google Cloud with two threads, it takes about 5 hours to build Sage from scratch, run rudimentary tests on the docker images, and upload them to Docker Hub and GitLab's registry.

<img src="gitlab.png" width=640, center, link=https:https://user-images.githubusercontent.com/373765/216875638-f961f94b-b99e-46a6-9129-fb44886fb26b.png.com/saraedum/sage/pipelines>

Recycling the build artifacts from the last run on the develop branch brings this down to about ?? minutes (on GitLab's free shared runners with two threads.) This roughly breaks down as:

• 32 minutes for `build-from-latest:
  • 10 minutes for the actual build (most of which is spent in the docbuild; caused by a known Sphinx bug to some extent)
  • ?? minutes are spent pulling the sagemath-dev image from Docker Hub (this usually goes away if you provision your own runners and expose the host's docker daemon by setting DOCKER_HOST.)
  • a few minutes running through all the fast stages of the Dockerfile.
  • a few minutes to push the resulting images to GitLab's registry. (using GitLab's cache, this could probably be improved, at least for runners that we provision ourselves.)
• 5 - 15 minutes for each test (run in parallel,); the relevant test is test-dev.sh which spents 6 minutes in the actual docbuild (just as in build-from-latest) and some 5 minutes to pull the sagemath-dev image from the GitLab registry. (That part should go away with a provisioned runner that exposes the host's docker daemon.)
• ?? minutes for the publishing to Docker Hub, most of which is spent pulling the images from the GitLab registry, and the other half pushing them to Docker Hub roughly. (Again, exposing the host's docker daemon would probably cut that time in half.)

With some tricks we could probably bring this down to 25 minutes (see CircleCI below) but we won't get this down to this without giving up on the CI being split up into different stages (as is for technical reasons necessary for CircleCI.) To go well below that, we would need to pull binary packages from somewhere…I don't see a sustainable way of doing this with the current SPKG system.

<img src="gitlab-rebuild.png" width=640, center, link=https:https://user-images.githubusercontent.com/373765/216875638-f961f94b-b99e-46a6-9129-fb44886fb26b.png.com/saraedum/sage/pipelines/18026318>

CircleCI

It typically takes almost 5 hours to build Sage from scratch on CircleCI, run rudimentary tests on the docker images, and upload them to Docker Hub.

<img src="circleci.png" width=640, center, link=https:https://user-images.githubusercontent.com/373765/216875640-484f5bb4-34be-4110-b4e6-7677ed57ac53.png.com/gh/saraedum/workflows/sage>

Recycling the build artifacts from the last run on the develop branch brings this down to about 30 minutes usually. 5 minutes could be saved by not testing the sagemath-dev and probably another minute or two if we do not build it at all. To go significantly below 15 minutes is probably hard with the huge sage-the-distribution (7GB uncompressed/2GB compressed) that we have to pull every time at the moment.

<img src="circleci-rebuild.png" width=640, center, link=https:https://user-images.githubusercontent.com/373765/216875640-484f5bb4-34be-4110-b4e6-7677ed57ac53.png.com/gh/saraedum/workflows/sage>

Docker Hub

A push to github updates the README on the Docker Hub page. The current sizes are and ; unfortunately MicroBadger is somewhat unstable so these numbers are incorrectly reported as 0 sometimes.

<img src="dockerhub.png" width=640, center, link=https://hub.docker.com/r/sagemath/sagemath>

---

Here are some things that we need to test before merging this:

• build-from-clean works in the sagemath namespace, building a tag on GitLab, https:https://user-images.githubusercontent.com/373765/216875638-f961f94b-b99e-46a6-9129-fb44886fb26b.png.com/saraedum/sage/pipelines/25831229
• build-from-clean works in the sagemath namespace, building from develop on GitLab, https:https://user-images.githubusercontent.com/373765/216875638-f961f94b-b99e-46a6-9129-fb44886fb26b.png.com/saraedum/sage/pipelines/25831675
• build-from-clean works in a user namespace on CircleCI, https:https://user-images.githubusercontent.com/373765/216875640-484f5bb4-34be-4110-b4e6-7677ed57ac53.png.com/workflow-run/4ae6af8c-2212-4724-a865-a401be4bd8b7; this does not work reliably as it often times out after 5 hours. If we can manage to use more packages from the system, then we should be able to move this below CircleCI's time limit.
• build-from-latest works and is fast in a user namespace on GitLab, https:https://user-images.githubusercontent.com/373765/216875638-f961f94b-b99e-46a6-9129-fb44886fb26b.png.com/saraedum/sage/pipelines/25894653
• build-from-latest works and is fast in a user namespace on CircleCI, https:https://user-images.githubusercontent.com/373765/216875640-484f5bb4-34be-4110-b4e6-7677ed57ac53.png.com/workflow-run/5bad5fe0-f817-4174-b0b4-de7d1be3b01c

---

After this ticket has been merged, the following steps are necessary:

• Setup an account for sagemath on Circle CI.
• Add Docker Hub credentials on Circle CI or GitLab.

To see a demo of what the result looks like, go to https://hub.docker.com/r/sagemath/sagemath/. The CircleCI runs can be seen here https:https://user-images.githubusercontent.com/373765/216875640-484f5bb4-34be-4110-b4e6-7677ed57ac53.png.com/gh/saraedum/sage, and the GitLab CI runs are here https:https://user-images.githubusercontent.com/373765/216875638-f961f94b-b99e-46a6-9129-fb44886fb26b.png.com/saraedum/sage/pipelines.

---

¹: I want to run unit tests of an external Sage package, https://github.com/swewers/MCLF. Being able to build a custom docker image which contains some not-yet-merged tickets makes this much easier.

PS: Long-term one could imagine this to be the first step to replace the patchbot with a solution that we do not have to maintain so much ourselves, such as gitlab-runners. This is of course outside of the scope of this ticket but having a bunch of working CI files in our repository might inspire people to script some other tasks in a reproducible and standardized way.

Depends on #25161
Depends on #25160

CC: @roed314 @embray @nthiery @mkoeppe @jdemeyer @hivert

Component: distribution

Keywords: CI

Author: Julian Rüth

Branch: ac6201a

Reviewer: Erik Bray

Issue created by migration from https://trac.sagemath.org/ticket/24655

[saraedum]

Description changed:

--- 
+++ 
@@ -14,4 +14,6 @@
 
 See also https://github.com/sagemath/docker-images/issues/13.
 
+To see a demo of what the result looks like, go to https://hub.docker.com/r/saraedum/sagemath/. The CircleCI runs can be seen here https://circleci.com/gh/saraedum/sage, and the [GitLab](../wiki/GitLab) CI runs are here https://gitlab.com/saraedum/sage/pipelines.
+
 ¹: I want to run unit tests of an external Sage package, https://github.com/swewers/MCLF. Being able to build a custom docker image which contains some not-yet-merged tickets makes this much easier.

[saraedum]

Branch: u/saraedum/gitlabci

[saraedum]

New commits:

91e4ed1

Enablec [GitLab](../wiki/GitLab) CI and CircleCI

[saraedum]

Description changed:

--- 
+++ 
@@ -17,3 +17,5 @@
 To see a demo of what the result looks like, go to https://hub.docker.com/r/saraedum/sagemath/. The CircleCI runs can be seen here https://circleci.com/gh/saraedum/sage, and the [GitLab](../wiki/GitLab) CI runs are here https://gitlab.com/saraedum/sage/pipelines.
 
 ¹: I want to run unit tests of an external Sage package, https://github.com/swewers/MCLF. Being able to build a custom docker image which contains some not-yet-merged tickets makes this much easier.
+
+PS: Long-term one could imagine this to be the first step to replace the patchbot with a solution that we do not have to maintain so much ourselves, such as gitlab-runners. This is of course outside of the scope of this ticket but having a bunch of working CI files in our repository might inspire people to script some other tasks in a reproducible and standardized way.

[saraedum]

Commit: 91e4ed1

[nthiery]

comment:4

I haven't played much with GitLab CI and CircleCI, so can't really comment on the details for now. But +1 for the general idea. That would be very helpful.

[nthiery]

comment:5

Adding Matthias who did setup continuous integration for various Sage packages (including https://github.com/sagemath/sage_sample).

[sagetrac-git]

Changed commit from 91e4ed1 to 18970ac

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

18970ac

Enable [GitLab](../wiki/GitLab) CI and CircleCI

[sagetrac-git]

Changed commit from 18970ac to 3c85557