Organize cmdeploy into install, configure, and activate stages #695
Conversation
cliffmccarthy
commented
Oct 26, 2025
cliffmccarthy
commented
- Added a class hierarchy of Deployer objects which implement install(), configure(), and activate() methods for each of the components that comprise a chatmail server.
- This is intended to provide the same functionality as the current process, though the order of the individual steps has been changed to organize them into the three separate stages of deployment.
- For more information, see the individual commit messages. The commits are intended to be individually reviewable logical steps in the transformation of the code. Also see the discussion of the new code structure in ARCHITECTURE.md.
- So far, I have tested this code on fresh installs and some upgrade scenarios, but I have not tested every possible upgrade path from past releases of chatmail/relay.
- Moved the "Add 9.9.9.9 to resolv.conf" step earlier, before the creation of users or updates to any config files. This should not affect any of those operations. Moving this step earlier makes it easier to accommodate the restructuring of the deployment process into separate components with separate stages for install, configure, and activate.
- Added a Deployer class that defines the base for objects that will handle installation of individual components, with install, configure, and activate stages. Subclasses will override the implementation methods of those stages as needed, while the base class handles all the logic of deciding which stages to execute. - The CMDEPLOY_STAGES environment variable is used to determine what stages to run. If this is not defined, all stages run as usual. - Added import of Deployer to cmdeploy/__init__.py. This is not yet used, but the next series of commits will use it. - In deploy_chatmail(), define an empty list of deployers, and call the create_groups() and create_users() methods for the items in the list. This list will get filled with Deployer objects in the next series of commits.
- Removed now-unused 'debug' variable from deploy_chatmail().
- Use policy-rc.d during nginx install. This is needed to keep nginx
from starting up and interfering with acmetool. For more information see:
- https://serverfault.com/questions/861583/how-to-stop-nginx-from-being-automatically-started-on-install
- https://major.io/p/install-debian-packages-without-starting-daemons/
- https://people.debian.org/~hmh/invokerc.d-policyrc.d-specification.txt
- Note that this moves the installation of the opendkim package earlier in the deployment sequence. Previously, it was installed during the _configure_opendkim() routine.
- This splits the existing deploy_iroh_relay() routine into methods for the install, configure, and activate stages.
- This splits the existing deploy_acmetool() routine into methods for the install, configure, and activate stages.
- This splits the existing deploy_mtail() routine into methods for the install, configure, and activate stages.
- This splits the existing _uninstall_mta_sts_daemon() routine into methods for the configure and activate stages.
- This replaces the existing _remove_rspamd() routine with a method for the install stage.
- Split _install_remote_venv_with_chatmaild() into three routines, to handle the install, configure, and activate stages. - This moves the upload of chatmail.ini later in the deployment process, because it is a configuration file specific to the instance, not software installation that would be uniform across all deployments.
- This moves the installation of cron earlier in the deployment sequence.
- This class is a special case because it has a dependency on the Postfix and Dovecot deployers. When deciding whether to restart the echobot service, it needs to know whether the Postfix and Dovecot deployers restarted their services. To support this dependency, the PostfixDeployer and DovecotDeployer objects are passed to the EchobotDeployer object, so it can check their was_restarted attributes.
- This adds a step to create /var/www in the install stage, because the directory needs to exist for the rsync in the configure stage to work.
- This splits the existing deploy_turn_server() routine into methods for the install, configure, and activate stages.
- The 'curl' program is used in TurnDeployer and IrohDeployer, so it makes more sense to install it at the beginning in ChatmailDeployer, rather than have each thing that uses it install it separately.
- The previous commits that added Deployer classes mostly kept deployment operations in the same order that they were in before. To organize the process into separate stages for install, configure, and activate, we need to reorder the method calls. This is the commit that does that, and thus this is the commit that has the largest effect on the order of operations. - The calls for the deployer objects are all reordered here so that the methods are called in the same sequence for each stage. This will allow us to collect the calls into loops in the next commit. This commit provides a way to see a diff showing exactly how the sequence changed. - The sequence of deployers was largely based on preserving the order of the "activate" stage, as this seems like the place order might be the most likely to matter. Installation of packages and configuration of files should generally be able to run in any order. (ChatmailDeployer handles updating the apt data, and therefore needs to be first, however.)
- Revised deploy_chatmail() to use all_deployers to call the install(), configure(), and activate() methods on all the deployers, rather than listing them explicitly in the code.
- Updated ARCHITECTURE.md to describe the Deployer class hierarchy and the motivations behind it.
There was a problem hiding this comment.
Love it! It makes the whole structure much clearer, and the commit structure made it very pleasant to review.
I have some small suggestions, none of them critical.
I like how it stays inconsistent whether a Deployer class is definied in cmdeploy/__init__.py or in a subdirectory (with acmetool). I think many of the Deployers could be in subdirectories, because they are pretty isolated, like turn, iroh, opendkim, journald, and mtail. For some it might make sense to have them in __init__.py because you can see at one glance whether they interact with other services (like acmetool, actually?).
I started the CI in #696, so we can notice errors, just in case.
We should probably have more than just my review for this, as it changes the structure deeply? @hpk42 @link2xt
| files.put( | ||
| name="Add Deltachat OBS GPG key to apt keyring", | ||
| src=importlib.resources.files(__package__).joinpath( | ||
| "obs-home-deltachat.gpg" | ||
| ), | ||
| dest="/etc/apt/keyrings/obs-home-deltachat.gpg", | ||
| user="root", | ||
| group="root", | ||
| mode="644", | ||
| ) |
There was a problem hiding this comment.
This can be completely removed as we don't install dovecot from OBS anymore :) We don't need to set present=False imo because the file doesn't hurt either.
| nginx_deployer, | ||
| rspamd_deployer, | ||
| fcgiwrap_deployer, |
There was a problem hiding this comment.
| nginx_deployer, | |
| rspamd_deployer, | |
| fcgiwrap_deployer, | |
| fcgiwrap_deployer, | |
| nginx_deployer, | |
| rspamd_deployer, |
I think we should activate fcgiwrap before nginx actually, otherwise there might be a short time where nginx routes to the fcgiwrap port but nothing listens.
| chatmail_deployer, | ||
| turn_deployer, | ||
| unbound_deployer, |
There was a problem hiding this comment.
| chatmail_deployer, | |
| turn_deployer, | |
| unbound_deployer, | |
| chatmail_deployer, | |
| unbound_deployer, | |
| turn_deployer, |
My head finds it more logical to do unbound before turn deployment, as DNS resolving is more system preparation.
Having turn being deployed next to iroh-relay makes sense as well.
I don't see a technical necessity to do it either way though.
| rspamd_deployer, | ||
| fcgiwrap_deployer, | ||
| echobot_deployer, | ||
| journald_deployer, |
There was a problem hiding this comment.
I'm a bit surprised journald is activated so late in the process - isn't logging important to the entire process, too? I think it should be directly before or after chatmail_deployer.
| ## Chatmail Server | ||
|
|
||
| This diagram shows components of the chatmail server; this is a draft |
There was a problem hiding this comment.
| ## Chatmail Server | |
| This diagram shows components of the chatmail server; this is a draft | |
| # Chatmail Relay | |
| This diagram shows components of the chatmail relay server; this is a draft |
Let's use chatmail relay or chatmail relay server ;) it does in some situations make sense to say "chatmail relay server" to clarify that it runs on servers; In general we like to say "relay" instead of "server" to point out that it doesn't really store messages and only relays them.
|
CI fails to download iroh-relay on the ipv4-only runner o.0 this did not happen before. Any ideas? Maybe unbound needs to be activated before the installation step? This would mess with the general concept of course. |
The "Download chatmail-turn" step runs almost the exact same thing and it worked in that run. The only thing that happened between that step and "Download iroh-relay" was installation of unbound, so that does suggest that unbound has something to do with it. I wonder if we can do the same thing I did with nginx to prevent the service from starting up as soon as it is installed. |
|
Ah, then unbound is out of the box simply misconfigured for an ipv4 only host? |
- On an IPv4-only system, if unbound is started but not configured, it causes subsequent steps to fail to resolve hosts. - Revised UnboundDeployer.install_impl() to use policy-rc.d to prevent the service from starting when installed. This is the same mechanism used to keep nginx from starting on install.
That seems possible, given what we have observed here. I haven't investigated how it works very deeply. I made the revision to use policy-rc.d like with nginx, and it works fine on a fresh install. You can cherry-pick commit 32b0472 onto the CI pull request to see if that solves it for the IPv4 system. |
2 participants
