Skip to content

tests: Add a sysext and confext test with a separate, empty /var - #2658

Open
jmarrero-bot wants to merge 1 commit into
bootc-dev:mainfrom
jmarrero-forge:bot/sysext-cs9-test
Open

jmarrero-bot wants to merge 1 commit into
bootc-dev:mainfrom
jmarrero-forge:bot/sysext-cs9-test

Conversation

@jmarrero-bot

Copy link
Copy Markdown
Contributor

This adds a tmt test that stacks a directory sysext (over /usr) and a directory confext (over /etc) on a bootc host whose /var is a separate partition. The partition is empty when the host first boots from it, and it holds the extensions.

The test merges and unmerges each extension from a shell. It then replaces a merged extension with a new version and checks that refresh shows the new one. Finally it checks that systemd-sysext.service and systemd-confext.service merge them at boot from the separate /var, and that no unit has failed once that boot has settled. Each failure starts with the component that broke: [layout], [bootc], [systemd-sysext], [systemd-confext] or [test harness].

The change touches only the test file and the regenerated tmt lists. Booting a disk installed inside the VM isn't possible here, and root fills the rest of the VM's disk. So, like the XBOOTLDR test, the test shrinks the ESP, which holds about 8M of its 512M, to 128M. It puts the /var partition in the space that frees and mounts it from /etc/fstab. Only tmt's plan directory and scripts, taken from tmt's environment, are copied onto it.

The test runs on CentOS Stream 9 and 10 in image mode, on the ostree backend only. That is one VM, with three boots, in one matrix cell per distribution. systemd-confext only exists from systemd 254 on, so CentOS Stream 9 (systemd 252) tests the sysext and logs a SKIP for the confext. On newer systemd, a missing systemd-confext fails the test.

Finding: on CentOS Stream 9, merging relabels /usr (systemd/systemd#34387, fixed in 257 by systemd/systemd#35132):

  • After a merge from a shell, /usr is user_tmp_t, and sshd refuses all new logins with chroot("/usr/share/empty.sshd"): Permission denied.
  • After a merge by the service, /usr is tmpfs_t.

On systemd older than 257, the test reports exactly the label each way of merging gives as a KNOWN ISSUE, and fails on any other label change. It unmerges before tmt has to log in again. CentOS Stream 10's systemd 257 keeps the /usr and /etc labels, so the check is strict there. tmt has no result that warns without failing the plan, so the known issue only shows in the test's output. A RHEL backport would keep systemd at 252, so a TODO next to the version gate says to drop the exception then.

Finding: a merged confext keeps /etc read-only for the whole boot. On CentOS Stream 10, bootc's own bootc-publish-rhsm-facts.service then fails with Writing etc/rhsm/facts/bootc.facts: Read-only file system. The test reports that one unit as a KNOWN ISSUE until bootc skips it on a read-only /etc, and fails on any other unit that fails.

Not covered: bootc upgrade or switch with extensions merged.

Testing was on a 16-core devspace (bcvk 0.19.0, tmt 1.80.0). Images were built like CI's ostree cells, at this tree. Logs are in https://gist.github.com/jmarrero-bot/ebbba1edfa9d7a805f304a789afec21b.

  • just test-tmt-nobuild extensions-separate-var passed on centos-9 (sysext, with the label KNOWN ISSUE lines) and on centos-10 (sysext and confext, with the rhsm KNOWN ISSUE).
  • These deliberately broken variants each failed with a tagged message:
    • sysext extension-release, on both distros: [systemd-sysext] ... is not visible under /usr.
    • confext extension-release, on c10: [systemd-confext] ... is not visible under /etc.
    • replacing the extension without a refresh, on both: [systemd-sysext] ... holds '... v1', expected '... v2'.
    • systemd-confext treated as missing, on c10: [systemd-confext] systemd 257 should have systemd-confext.
    • a unit that fails 60s into the final boot, on c10: [systemd-confext] late-fail-demo.service failed ....
  • just validate passed.

Part of jmarrero-forge/tracker#88. The docs half is a separate PR.

The Signed-off-by: Joseph Marrero Corchado <jmarrero@redhat.com> on these commits was added on jmarrero's approval of the review draft: jmarrero-forge#22 (comment)

Generated-by: https://github.com/jmarrero-bot#llms

Hosts that adopt bootc on their existing layout will stack system and
configuration extensions on it, both on the RHEL 9 long tail and on
RHEL 10, so CI should show when that breaks, and which project broke
it.

The test stacks a directory sysext (over /usr) and confext (over /etc)
on the image under test, on a host whose /var is its own partition,
empty when the host first boots from it, and which holds the
extensions. It merges and unmerges each from a shell, refreshes a
merged one after replacing it with a new version, then has
systemd-sysext.service and systemd-confext.service merge them at boot
from the separate /var, and checks that no unit failed once that boot
settled.
Booting a disk installed inside the VM
isn't possible here, and root fills the rest of the VM's disk, so like
the XBOOTLDR test it shrinks the ESP, which holds about 8M of its 512M,
and puts the /var partition in the space that frees, mounted from
/etc/fstab, with only tmt's own files copied onto it. Each failure
names the component that broke: the layout, bootc, systemd-sysext,
systemd-confext or the test harness.

It runs on CentOS Stream 9 and 10, on the ostree backend only, so it
adds one VM to one matrix cell per distribution. systemd-confext only
exists from systemd 254 on, so on CentOS Stream 9 (systemd 252) only
the sysext is tested; on newer systemd a missing systemd-confext fails.

On CentOS Stream 9 merging relabels /usr (systemd/systemd#34387, fixed
in systemd 257): to user_tmp_t when merged from a shell, after which
sshd refuses logins, and to tmpfs_t when merged by the service. On
systemd older than 257 the test reports the label each way of merging
gives as a known issue, and fails on anything else; CentOS Stream 10's
systemd 257 keeps the labels, so it is strict there. It unmerges
before tmt has to log in again. Neither tmt nor the harness has a
result that warns without failing the plan, so the known issue is only
in the test's output.

A merged confext keeps /etc read-only for the whole boot, and on
CentOS Stream 10 bootc-publish-rhsm-facts.service then fails writing
/etc/rhsm/facts/bootc.facts. The test reports that one unit as a known
issue for now, until bootc skips it on a read-only /etc.

Generated-by: AI
Signed-off-by: Joseph Marrero Corchado <jmarrero@redhat.com>
@jmarrero-bot

Copy link
Copy Markdown
Contributor Author

Rebased onto main; 1 commit, no content change.

Generated-by: https://github.com/jmarrero-bot#llms

@jmarrero-bot

Copy link
Copy Markdown
Contributor Author

Rebased onto main to rerun CI, with no content change. The fedora-44 composefs ext4 grub-cc cell had failed because the VM for plan-01-readonly never became ready (an infrastructure failure), which isn't related to this test. jmarrero's sign-off is kept on 5a7a9b8.

Generated-by: https://github.com/jmarrero-bot#llms

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant