Skip to content

apply-live: Bound images live apply - #2659

Draft
yasminvalim wants to merge 8 commits into
bootc-dev:mainfrom
yasminvalim:apply-live-images
Draft

yasminvalim wants to merge 8 commits into
bootc-dev:mainfrom
yasminvalim:apply-live-images

Conversation

@yasminvalim

Copy link
Copy Markdown

Summary

Rebase and cleanup of the apply-live bound-images work from #2458, incorporating the follow-up fixes from jmarrero-forge#10 (fast path, unit reconciliation, override checks), plus review follow-ups: soft-reboot clear only at --apply handoff, and dropping liveBoundImages from status/schema in favor of /run markers.

Test plan

  • cargo test -p bootc-lib --lib
  • just test-tmt logically-bound-apply-live

jmarrero and others added 8 commits November 13, 2026 15:03
Prep for `bootc apply-live bound-images`, which needs to know not just
which image a bound image definition references, but where the quadlet
file that defines it lives (so it can be related to podman's quadlet
search directories) and its contents (so definitions can be compared
across deployments and materialized elsewhere).

Add `query_bound_image_specs` returning that alongside the parsed
`BoundImage`, and make `query_bound_images` a thin wrapper over it.
Symlink targets are resolved lexically to a root-relative path with the
same semantics as the `RESOLVE_IN_ROOT` read of the file (excess `..`
stays at the root); no further symlinks are followed.

Assisted-by: AI
A customer using logically bound images wants to update them out of
band without rebooting, while bootc keeps owning their lifecycle
(storage, garbage collection, rollback). This is a scoped first step
toward the general `bootc apply-live` in bootc-dev#76.

The key observation is that the image bytes are already live: there is
a single bootc-owned containers-storage shared by all deployments, and
quadlets read it via `additionalimagestore`. What is tied to a
deployment is only the definition (the `bound-images.d` symlink and the
quadlet it points to), which the running system reads from its own
`/usr`. So no overmounting is needed: changed quadlets are written to
`/run/containers/systemd/`, which podman gives precedence over `/etc`
and `/usr`, systemd is reloaded, and the affected `.container` units
restarted. Because `/run` is transient the override disappears on
reboot, at which point the staged deployment's content applies and the
system converges on its own. A soft reboot preserves `/run`, so the
override is cleared when one is prepared.

The command refuses to run unless the ostree diff between the booted
and staged commits is entirely accounted for by the bound image changes
it will apply, so it never silently applies half an upgrade; since the
diff doesn't enumerate added/removed directories, those are walked in
the deployment roots. The applied state is recorded in
`/run/bootc/apply-live/bound-images.json` before any unit is touched,
with each unit marked pending until its restart succeeds, so a failed
restart (or `--no-restart`) is retried by simply re-running. The state
is surfaced in `bootc status`, and its image names become extra GC
roots so a `bootc rollback` that discards the staged deployment doesn't
prune an image a running container uses. A later staged deployment
reverts any overrides from a previous apply that it no longer needs.

The verb is a subcommand group (`apply-live bound-images`, alias `lbi`)
so that the bare `apply-live` stays reserved for the general feature and
further scopes can be added without a new top-level verb. It is hidden
and experimental, and ostree-only for now; composefs needs a tree diff
API before the scope check can be implemented there.

This has unit coverage for the planning, scope check and state logic.
The /run write, SELinux labeling, systemctl sequencing, status output,
GC root handling, out-of-scope refusal and post-reboot convergence were
verified on a CentOS Stream 10 VM by the tmt test in the following
commit.

Assisted-by: AI
The unit tests cover the planning, scope check and state file logic,
but not the parts that only exist on a real host: writing to
`/run/containers/systemd` with the right SELinux label, the systemctl
sequencing, and the interaction with `bootc status`, garbage collection
and reboot.

Switch to an image with a bound `.container` that just sleeps, then
upgrade to one that only changes the bound image, and walk through
`--dry-run`, `--no-restart` (unit still on the old image, state marked
pending), a plain re-run (unit restarted onto the new image), and the
idempotent no-op. Discarding the staged deployment via rollback and
running cleanup checks the live state acts as a GC root. An upgrade
with an unrelated file change checks the out-of-scope refusal. After
the reboot, the `/run` override and state must be gone and the
deployment's own definition in effect.

Assisted-by: AI
Apply two pieces of feedback from cgwalters's review of bootc-dev#2458:

- Scope the module doc's "doesn't require mounting anything" claim to
  the ostree backend, since the composefs backend lays out image
  storage differently.

- Replace the `/run/bootc/apply-live/bound-images.json` state file
  with state embedded directly in the quadlet files written to
  `/run/containers/systemd/`: each one now carries a leading marker
  comment recording its staged checksum, image reference, and whether
  its unit still needs a restart. That directory is now the source of
  truth instead of a bespoke JSON file. A small flat-text list of
  protected image references is still written separately, since a
  bound image with an unchanged definition has no quadlet override to
  carry that state, and this list is only used to protect such images
  from garbage collection.

Also address several correctness issues GitHub Copilot's automated
review found in that state-tracking code:

- Track the previous unit name across a `ServiceName=` change so the
  old unit is actually stopped instead of left running alongside the
  new one.
- Merge newly staged protected images into the existing
  `protected-images.list` instead of replacing it, so an earlier
  deployment's still-running bound image isn't GC'd out from under it.
- Make the fast path compare the full set of expected quadlets, not
  just checksums, so a retry after a partially-failed apply doesn't
  mistake incomplete state for complete.
- Refuse to overwrite a `/run` quadlet override bootc didn't write
  itself.
- Guard against an Added/Removed pair that target the same `/run`
  destination (e.g. a quadlet moving between search dirs) from
  deleting the replacement that was just written.
- Always refresh a quadlet's marker checksum, even when its contents
  are unchanged, so status and the fast path stay accurate across
  later deployments.
- Skip symlinks and other non-regular files when scanning
  `/run/containers/systemd/`, so a unit masked via `-> /dev/null`
  doesn't break `bootc status`.
- Update the booted integration test to read state from `bootc status
  --json`'s `liveBoundImages` and `protected-images.list` instead of
  the now-removed JSON state file.

Still under review; not yet addressing the separate comment about
whether `liveBoundImages` belongs in `bootc status`/the schema, nor
the deeper `/etc`-vs-vendor quadlet precedence question Copilot also
raised.

Assisted-by: AI
Reuse the apply-live fixes from jmarrero-forge#10 (0881235,
00b527a, b428ed4, 636c124). Cherry-pick conflicted against the
marker-based tip, so those commit trees are adopted here:

- Gate the fast path on applied-checksum; always run the scope check
- Plan stops/restarts from live state (pending-restarts / stopped-units)
- Refuse non-bootc /run overrides and /etc-shadowed vendor quadlets
- Restart units an earlier live removal stopped when they return
- Extend the booted tmt coverage for those cases

Generated-by: AI
Preparing soft-reboot without --apply leaves containers running; clearing
/run overrides and protected images then lets GC prune in-use images and
drops definitions while units still use them. Clear only when --apply
triggers the reboot handoff.

Assisted-by: AI
Joseph asked whether live-applied bound images belong in HostStatus /
the schema. Live state already lives in /run markers
(/run/containers/systemd and /run/bootc/apply-live); surfacing it in
status duplicates that and couples the schema to an experimental
feature. Drop the types, status wiring, schema defs, and point the
booted tmt test at the markers instead.

Assisted-by: AI
Signed-off-by: Yasmin de Souza <ydesouza@redhat.com>
Number 49 is already taken by test-49-composefs-1-16-bridge; the
generated fmf already registered this test as 68. Align the # number
comment so `cargo xtask update-generated direct --check` passes.

Assisted-by: AI
@yasminvalim

Copy link
Copy Markdown
Author

@jmarrero I am not sure why but I am unable to run the CI. Can you run it for me?
Thank you!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/documentation Updates to the documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants