Skip to content

composefs: Stale ESPs on multi-disk roots make GC delete newer deployments #2653

Open

Description

@fcuny

On the composefs backend, bootc writes boot artifacts to a single ESP. When the root sits on a multi-disk md RAID1 array, every other disk's ESP goes stale after the first upgrade. This causes two problems:

  1. If the machine boots from a stale ESP, GC treats every deployment missing from that ESP as orphaned. It deletes their state directories and prunes their images from the shared root.
  2. If the first disk dies, the machine silently boots the install image, several upgrades behind.

I'd like to agree on an approach here before sending a PR.

Setup

  • arm64 VM (KVM, edk2, Secure Boot off) with two disks and no NVRAM boot entries. The firmware boots EFI/BOOT/BOOTAA64.EFI from the first disk that has one.
  • Each disk has its own GPT table and ESP (sda2, sdb2). The ESPs sit outside any RAID.
  • The root filesystem is on an md RAID1 array across one partition on each disk. /var is on a second RAID1 array.
  • Installed with bootc install to-existing-root --composefs-backend. systemd-boot sits at the removable path on both ESPs, with UKIs. No XBOOTLDR partition.
  • After the install, our installer copies the ESP that bootc wrote onto the other disk's ESP, once.
  • Tested with bootc 1.17.3. The code at 0c5dd06 takes the same paths (see below).

Reproducer

Three images: A (installed), B and C. Steps 1 to 5 and step 6 come from two runs of the same test.

  1. Install A. Both ESPs match after the installer's copy.
  2. bootc switch to B. bootc writes B's UKI and loader/entries.staged/ to the booted ESP (sda2) only.
  3. Reboot. Disk 0 boots B. sda2 has entries for B and A. sdb2 still has only A.
  4. bootc switch to C and reboot. Disk 0 boots C. sda2 holds C and B, and GC removed A's UKI and entry from it. sdb2 still holds only A.
  5. Blank disk 0 and boot. The firmware falls back to sdb2, and the machine boots A, two switches behind. bootc status shows no rollback. Nothing reports an error.
  6. While booted into A from sdb2, run bootc internals composefs-gc --prune-repo. /sysroot/state/deploy holds A, B and C before. With RUST_LOG=debug, GC logs bootloader_entries containing only A, then cleans up B's and C's state directories as orphaned. It reports Objects: 107 removed (649594714 bytes) and Pruned symlinks: 8 images, 39 streams. Afterwards, /sysroot/state/deploy holds only A.

After step 6, B and C are gone along with their images. Replacing disk 0 and resyncing the array can't bring them back.

Why

Line references are at 0c5dd06.

bootc uses one ESP.

  • Install takes the first ESP among the root's backing disks: find_first_colocated_esp() in boot.rs:866 and boot.rs:1909. MountedImageRoot::new also stops at the first ESP and carries a TODO for this (boot.rs:2212).
  • A booted system resolves one ESP from LoaderDevicePartUUID in find_booted_from_esp() (store/mod.rs:506). Storage holds a single BootPartition built from it (store/mod.rs:619). Upgrade, switch, rollback, finalize and GC all go through it.
  • Discovery already finds every ESP. find_colocated_esps() walks the root's backing disks only (blockdev.rs:196). The writers just use one of them.

GC trusts that one ESP. GC takes the set of live deployments from the entries on the booted ESP (gc.rs:302, gc.rs:549). It deletes every state directory without an entry there (gc.rs:648). The guard at gc.rs:640 only protects the booted deployment.

The rest of this section comes from reading the code. I haven't run it.

  • Normal operation reaches the same GC. Step 6 used the internals command. Any bootc upgrade or bootc switch that stages a deployment runs the same GC with repo pruning (update.rs:511). The GC at finalize skips pruning, and state-directory cleanup only runs with pruning (gc.rs:589). That explains why A's state directory survived step 4 on sda2.
  • A healthy array is exposed too. It only takes one boot from the second disk, for example after a firmware update reorders boot entries or after disk 0 times out once.
  • Failure mode 2 has a time limit. On a healthy disk 0, the next upgrade after step 4 would delete A's state directory. From then on, sdb2 points at a deployment that no longer exists, and losing disk 0 leaves nothing bootable.

Proposal

  1. Write every ESP on the root's backing disks. This covers install, upgrade, switch, rollback and finalize. Storage would hold the set from find_colocated_esps() instead of a single BootPartition. #2645 moved Type 1 entry writes to one place, which makes the loop cheaper.
  2. Compute GC liveness from the union of entries across those ESPs, then clean each ESP. This keeps GC safe while the ESPs disagree, for example on an array that was degraded during an upgrade.
  3. Warn when an ESP is missing, and fail when a present ESP can't be written. A degraded array must still upgrade from its remaining disk. A present ESP that silently stays stale is the bug this fixes.
  4. Enable it by default. Discovery only covers the root's own disks, so a Windows ESP on another disk stays untouched (#2554). Single-disk systems see no change.

Point 2 matters even without point 1. On its own, it prevents the data loss in step 6.

Alternative: write one ESP, then sync the others

bootc would write the booted ESP as today. It would then sync the paths it owns (EFI/Linux/bootc, loader/entries, the systemd-boot binary) to the other ESPs, deletions included. This is a smaller change. It still has to pick a source of truth, though. After booting from the stale ESP, the sync would copy stale state over the good ESP. Writing every ESP and computing GC liveness from the union avoids that.

Questions

  1. Write every ESP, or write one and sync?
  2. Should the GC liveness fix land first, as its own PR?
  3. Should a present ESP that fails to write fail the operation, or warn?
  4. Default on, or opt-in through an install config key?
  5. With an XBOOTLDR partition (#2573), should the same rule apply per disk, writing each ESP and its sibling XBOOTLDR? I'd leave this out of the first PR.
  6. Should bootc status report ESPs that disagree? Also a follow-up, in my view.

Out of scope

The ostree backend has the same single-ESP TODO in mount_esp_part (bootloader.rs:47). bootupd manages ESP content there, so it's a separate problem.

Testing

  • A unit test pinning find_colocated_esps() to the root's backing disks.
  • A unit test for GC liveness across two boot directories that disagree.
  • An integration test needs a VM with two disks and a RAID1 root. Can bcvk or the tmt plans attach a second disk? If not, I'd welcome pointers on where such a test should live.

Assisted-by: AI

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions