Skip to content

generator: Create /var/lib/systemd early instead of reordering tmpfiles - #2628

Merged
cgwalters merged 2 commits into
bootc-dev:mainfrom
jmarrero-forge:bot/tmpfiles-ordering-cycle
Nov 6, 2026
Merged

cgwalters merged 2 commits into
bootc-dev:mainfrom
jmarrero-forge:bot/tmpfiles-ordering-cycle

Conversation

@jmarrero-bot

Copy link
Copy Markdown
Contributor

On native composefs boots, the generator's drop-in from 41049cc orders systemd-tmpfiles-setup.service (After=local-fs.target) before systemd-random-seed.service. With a crypttab swap keyed from /dev/urandom, local-fs.target needs random-seed:

local-fs.target -> tmp.mount (After=swap.target) -> dev-mapper-swap.swap
  -> systemd-cryptsetup@swap.service (After=systemd-random-seed.service)
  -> systemd-random-seed.service -> systemd-tmpfiles-setup.service (drop-in)
  -> local-fs.target

systemd breaks the cycle by deleting one of these jobs. On different boots that was random-seed (so it doesn't run at all), or tmp.mount (the report: tmpfiles runs before /tmp is mounted, GNOME can't start Xwayland). The reporter uses the native composefs backend (bootc-dev/bootc#2345), where the drop-in is written as intended; ostree boots never get it.

This replaces the drop-in with a generated bootc-early-tmpfiles.service, which runs systemd-tmpfiles --create --prefix=/var/lib/systemd. It is ordered after only what random-seed is ordered after itself (systemd-remount-fs.service and the mounts for /var/lib/systemd), so it can't add a cycle. The only tmpfiles.d lines under /var/lib/systemd in the CentOS Stream 9/10 and Fedora 45 base images are root-owned, so it doesn't need sysusers first. The PR also adds a tmt test that boots with a random-key encrypted swap (file-backed, in /var) and tmp.mount enabled, and checks that the journal has no ordering cycle and that tmp.mount was active before tmpfiles-setup started.

Testing, on a 16-core devspace (jmarrero-devspace-42012032986), all with CentOS Stream 10 images (systemd 257), composefs = BOOTC_variant=composefs, systemd-boot, ext4, BLS, unsealed:

  • Reproduction on main (1.17.4): the new tmt test fails. Its journal has local-fs.target: Found ordering cycle on tmp.mount/start; has dependency on swap.target/start, ... systemd-random-seed.service/start, systemd-tmpfiles-setup.service/start, local-fs.target/start and Job tmp.mount/start deleted to break ordering cycle. A manual bcvk VM from the same image deleted systemd-random-seed.service/start instead.
  • With this PR: just test-tmt-nobuild readonly encrypted-swap-ordering passes on composefs and on ostree (grub). On the composefs VM, bootc-early-tmpfiles.service ran before random-seed, the swap and /tmp came up, there were no AVC denials, and systemd-analyze verify default.target was clean.
  • The problem 41049cc fixed stays fixed. A control build of main without the drop-in fails readonly/014-test-no-failed-units with systemd-random-seed.service failed and avc: denied { add_name } for comm="systemd-random-" name="systemd" scontext=init_t tcontext=unlabeled_t. With this PR, 014 passes on composefs and on ostree.
  • just unit-tests (including the new generator::tests::test::test_early_tmpfiles_unit) and just validate pass. After review, a comment was added to the unit file; cargo test -p bootc-lib --lib generator passed again on that head, and systemd-analyze verify of the unit is clean in the CentOS Stream 9/10 and Fedora 45 images.

Not tested: Fedora images, UKI or sealed boots, and a TPM (systemd-tpm2-setup.service was skipped by its condition in these VMs).

Fixes #2624

The Signed-off-by: Joseph Marrero Corchado <jmarrero@redhat.com> on these commits was added on jmarrero's approval of the review draft: jmarrero-forge#8 (review)

Generated-by: https://github.com/jmarrero-bot#llms

On native composefs boots, 41049cc ordered all of
systemd-tmpfiles-setup.service before systemd-random-seed.service so
that /var/lib/systemd exists when random-seed needs it. But
tmpfiles-setup runs after local-fs.target, while random-seed is an early
unit that local-fs.target may itself need: a crypttab swap keyed from
/dev/urandom orders its systemd-cryptsetup@ unit after random-seed, and
tmp.mount is After=swap.target. That closes an ordering cycle, which
systemd breaks by deleting one of its jobs: depending on the boot,
random-seed doesn't run at all, or (as in the report) tmpfiles runs
before /tmp is mounted and GNOME can't start Xwayland.

Instead, generate a small unit that runs systemd-tmpfiles for just
/var/lib/systemd, ordered after only what random-seed is ordered after
itself (systemd-remount-fs and the mounts for /var/lib/systemd), so it
can't add a cycle that random-seed doesn't already have. Every
tmpfiles.d line under /var/lib/systemd in Fedora and CentOS is
root-owned, so it doesn't need systemd-sysusers first.

Fixes: 41049cc ("generator: Order tmpfiles before random seed and TPM setup")
Closes: bootc-dev#2624
Generated-by: AI
Signed-off-by: Joseph Marrero Corchado <jmarrero@redhat.com>
Regression test for the ordering cycle in bootc-dev#2624: set up
a crypttab swap keyed from /dev/urandom with tmp.mount enabled, reboot,
and check that systemd found no ordering cycle and mounted /tmp before
systemd-tmpfiles-setup started.

Generated-by: AI
Signed-off-by: Joseph Marrero Corchado <jmarrero@redhat.com>

@cgwalters cgwalters left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ultimately I think this needs to be fixed in systemd OR selinux-policy, AIUI it's basically just the (Fedora?) selinux policy denying systemd-random-seed to create those files.

But still yeah, would be cleaner to have an early tmpfiles there I think

@cgwalters
cgwalters enabled auto-merge (rebase) November 6, 2026 12:16
@cgwalters
cgwalters merged commit c3be524 into bootc-dev:main Nov 6, 2026
55 checks passed
@jmarrero-bot jmarrero-bot moved this from In Review to Done in Workstream Nov 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Nasty Regression in 1.17. Ordering cycle preventing Gnome login with Encrypted Swap

2 participants