test: initrd in /usr

[pothos]

Still WIP

How to use

Depends on flatcar/bootengine#110

Testing done

• Changelog entries added in the respective changelog/ directory (user-facing change, bug fix, security fix, update)
• Inspected CI output for image differences: /boot and /usr size, packages, list files for any missing binaries, kernel modules, config files, kernel modules, etc.

[pothos]

@chewi My idea is to load the "normal" initrd as loopback mount from /usr and switch to it for Ignition, network drivers and so on. If you want we could try to plug your busybox experiments in on this branch. To keep things simple and avoid risking breakage I would assume that the minimal initrd still ships the CPU microcode for the kernel to load because when we load it from userland there are more things to be aware of that aren't well supported, as far as I remember. Also, the /usr verity mount should be done by the minimal initrd and we would bind mount it into the "normal" initrd mount to reuse it (or maybe dm-verity also has no problems doing the work twice? For performance reasons we still might want to only do it once). In the end we could even drop the wrappers if we start to include afterburn and ignition (and other wrappers) again in the initrd and remove it from /usr where they aren't actually needed.

[github-actions]

Build action triggered: https://github.com/flatcar/scripts/actions/runs/17452085773

[chewi]

Although I shared the same concern about losing functionality that we would have to reimplement, I hadn't yet identified any such functionality, so I'm not quite ready to throw out my proposal to go straight from tiny initrd to real /usr. I'd really like to know what your specific concerns are.

This is an interesting approach in any case. My own alternative would have been to mount /usr as an overlay with the initrd, deleting all the duplicate files from the initrd, but I hadn't fully thought it through.

Regarding verity, I think it only needs to be set up once. I didn't enable verity in my own experiment, but /sysroot/usr was simply a bind mount of /usr. I think that would still work with verity applied.

[chewi]

Suggested change

	mkdir "${S}"/build/bootengine
	cat "${S}"/build/bootengine.cpio | while cpio --no-absolute-filenames -d -m -D "${S}"/build/bootengine -i ; do :; done
	# Lacking read permissions even for user
	chmod u+r "${S}"/build/bootengine/etc/gshadow
	# CPU microcode should stay in the minimal initrd
	rm -rf "${S}"/build/bootengine/kernel "${S}"/build/bootengine/early_cpio
	# TODO: rework /usr verity mount to reuse the one from the minimal initrd
	mksquashfs "${S}"/build/bootengine "${S}"/build/bootengine.img -all-root -noappend -xattrs-exclude ^btrfs.
	mkdir "${S}"/build/bootengine || die
	pushd "${S}"/build/bootengine || die
	sudo lsinitrd --unpack "${S}"/build/bootengine.cpio || die
	# TODO: rework /usr verity mount to reuse the one from the minimal initrd
	sudo mksquashfs . "${S}"/build/bootengine.img -noappend -xattrs-exclude ^btrfs. || die
	popd || die

This is running with RESTRICT="userpriv", so you can use sudo here. That will ensure the device nodes are created correctly. lsinitrd --unpack will not include the microcode.

[chewi]

Also, might as well use mkfs.erofs now!

[chewi]

With bsdtar (which is in the SDK), you can even avoid extracting if dracut is called with --no-compress --no-early-microcode.

bsdtar -c @"${S}"/build/bootengine.cpio | mkfs.erofs --tar=f "${S}"/build/bootengine.raw

[pothos]

Thanks for the improvements!

Also, might as well use mkfs.erofs now!

Yes, it's not part of the SDK yet.

[pothos]

my proposal to go straight from tiny initrd to real /usr. I'd really like to know what your specific concerns are.

My intention was to keep most things untouched so that we can focus on the bare task of jumping into the regular initrd and avoid any risk of reimplementing all needed initrd logic. Things that should run from the initrd are: Ignition stages, hostname setup with afterburn (and basic network setup for them while they prepare the final network setup for the real system), setup of the /etc overlay, A/B sysext setup for OEM and extra sysexts (inc. fallback download), encrypted rootfs unlocking, generation of host-specific /etc files, disk uuid init, first-boot detection, and propably other stuff I don't remember. I think we should rely on the current code for almost all of it to avoid breaking things.

[chewi]

Okay, but I wasn't proposing rewriting all that. Dracut puts those scripts into an initrd. I was just going to put them in /usr instead. It's more or less the same thing. It's the scripts that Dracut itself provides through its own modules that I was concerned about.

[pothos]

The question is on how these things are started because they run in a context with dependencies. Having only one set of systemd units for both the initrd and the final system doesn't work if we want to make use of systemd in the initrd - it would run all enabled units under /usr (unless we inject the initrd check into most of them). Creating a separate environment manually without dracut we would have more work and risk compared to moving the current initrd as a whole. When we get things working and have time left we can still try this out as optimization (preparing an /etc for the initrd stage with /etc/initrd-release, pulling in the ignition systemd units and so on, and with masking or customizing all unnecessary units through drop-ins).