Proposal: Restructure cmdeploy to facilitate container image builds

[cliffmccarthy]

To build container images for deployment under runtime environments such as Docker, we need an image, which contains the filesystem content that will be visible to processes running in the containers. Use of containers will facilitate swapping out individual components of the chatmail software environment in the future, but for an initial baseline implementation, we can aim to match the "software stack" of a server with host-based services under systemd. We already have a well-defined process for creating this stack, in the pyinfra-based cmdeploy code. Reusing this code to build an image provides the benefit of having a single process that produces matching results under both deployment methods, and that will stay aligned over time as the processes are changed.

A common practice in container development is to run one specific application in each container, and to make each image as small as possible, with only the content that container absolutely needs. For now, we do not have to aspire to that practice here. Instead, we can build a single image with many applications installed in it, that looks basically like the software stack of a host-based deploy. We can still run multiple containers with that image, using different entrypoint and command arguments to differentiate the containers into specific applications.

The current process implemented by cmdeploy performs three primary types of operation:

1. Installation of software, universal across all deployments.
2. Configuration of software, with deploy-specific variations.
3. Activation of services.

We can further divide the second item into:

• 2a. Configuration that is essential to the definition of a chatmail server and the same across all deployments.
• 2b. Configuration of "preferences" that vary among some deployments, but may be common across groups of deployments.
• 2c. Configuration of information that is completely unique to a single deployment, including anything with the server name baked in, and "secrets" such as private keys.

For a general-purpose container image, it is desirable to do as much as possible in the image build stage, incorporating everything that would be common across all deploys of that image. This would clearly include everything from item 1. Ideally, it would also include everything from item 2a, while the information for items 2b and 2c would be injected at runtime through arguments, environment variables, or file mounts. However, the applications in the stack do not generally divide their configurations along those boundaries so cleanly for us. This means that we probably will always need some step in deployment where application configuration files are derived from source-controlled templates, applying deploy-specific parameters to generate files like dovecot.conf and main.cf.

The current implementation of cmdeploy is not explicitly divided along these categories of operation. For example, postfix installation is implemented in deploy_chatmail(), with configuration in a separate _configure_postfix() routine, while the installation and configuration of mtail is all handled in a single deploy_mtail() routine. The steps are also not grouped into a clear "installation phase" and "configuration phase" in the main routine, with the order of installation, configuration, and activation somewhat interleaved across different applications. It is also not immediately evident whether there are steps where the operations must occur in the current order -- for example, whether there are any steps where configuration of one component must occur before installation of another.

To build images, it would help to be able to call everything in item 1, the basic installation steps, and nothing else. This could be achieved by sprinkling 'if' statements throughout the code, but that is not the best approach for maintainability or understandability. Instead, I propose we consider refactoring the installation process explicitly around the three main phases of installation, configuration, and activation, while maintaining functional equivalence to the current process, to ensure compatibility with existing host-based deploys.

Once we have this explicitly represented in code, it should be straightforward to provide image builds a way to call only the installation steps, while allowing host-based deploys to continue calling all the operations for standing up the whole server.

I have some work in progress towards this goal, and can put it in a pull request soon.

How to handle the configuration operations in a container environment is another whole topic to consider, and probably worth its own issue. The refactoring proposed here will at least make it easier to examine the configuration phase on its own, as we evaluate options for how it should best be handled in a container-based deploy.

The third phase, activation, is fairly straightforward once the other details have been worked out. It can be achieved with a compose file, the equivalent docker run commands, or something similar. The exact method will likely be subject to the preferences of the administrator of the server. If we build phases 1 and 2 with clear interfaces for deployment, this step will fall into place fairly easily.

[Simon-Laux]

would be nice to build docker image with nix to make it reproducible and extra efficient.
https://nix.dev/tutorials/nixos/building-and-running-docker-images.html#build-your-first-container

[cliffmccarthy]

would be nice to build docker image with nix to make it reproducible

Reproducibility is an excellent goal. All the current chatmail container work is focused on building containers using the same pyinfra-based programs that are used to deploy host-based server instances. This is an inherently mutable process; it calls out to shell commands and uses apt to fetch unpinned distro-level dependencies.

I think the restructuring proposed here supports your goal of having different ways to build images. By separating the construction of the software runtime environment from deploy-time actions, it becomes easier to consider alternative implementations. For example, installation of packages like postfix and dovecot should be separable from obtaining HTTPS certificates. With the right architecture to clearly delineate these stages of deployment, I think many options become open to us for changing the components within that architecture.

Thanks for your suggestion of an alternative mechanism that we can consider enabling through architectural flexibility!

[hpk42]

sounds like a good approach to me, especially keeping current pyinfra "cmdeploy run" invocations providing seamless compatibility.

I think the cmdeploy/__init__.py should be rather empty, and only import some things as needed from modules.
the current way i don't consider good practise in python FWIW.

Looking forward to seeing the PR addressing this whole issue :)

[missytake]

Yeah, I'd love if we could break up this monster __init__.py file. It's also annoying to find with tab-autocomplete. The directory structure with cmdeploy/src/cmdeploy and then you have to press / manually because there is also the cmdeploy.egg-info directory, it's super annoying for my auto-completion

[cliffmccarthy]

In my work so far, I kept everything in the existing file, but I could certainly use this opportunity to move things out of __init__.py. Would you rather see one file per deployed component, or continue to have them all in one file, just with a different name? (The deploy of acmetool is already in its own file; I could follow that pattern for all the other components. Note that this will be about 15 new files if I put each thing in its own file.)

[missytake]

Separate files for different components probably make it easier to find the correct function... for acmetool I think it's already the case even.

One large file makes it easier to read everything in one run though. Hm.

[cliffmccarthy]

Right, there are times lots of small files can be more awkward, and sometimes it's easier to spot patterns when things are in the same file. Breaking it up into more files will be pretty straightforward after the revisions I am preparing, so maybe we can look at how we want things to be organized as a later step.

[cliffmccarthy]

I submitted pull request #695 to implement the install / configure / activate structure proposed here.