Repository navigation
apply-live: Bound images live apply - #2659
Draft
yasminvalim wants to merge 8 commits into
Draft
yasminvalim wants to merge 8 commits into
yasminvalim wants to merge 8 commits into
Conversation
Prep for `bootc apply-live bound-images`, which needs to know not just which image a bound image definition references, but where the quadlet file that defines it lives (so it can be related to podman's quadlet search directories) and its contents (so definitions can be compared across deployments and materialized elsewhere). Add `query_bound_image_specs` returning that alongside the parsed `BoundImage`, and make `query_bound_images` a thin wrapper over it. Symlink targets are resolved lexically to a root-relative path with the same semantics as the `RESOLVE_IN_ROOT` read of the file (excess `..` stays at the root); no further symlinks are followed. Assisted-by: AI
A customer using logically bound images wants to update them out of band without rebooting, while bootc keeps owning their lifecycle (storage, garbage collection, rollback). This is a scoped first step toward the general `bootc apply-live` in bootc-dev#76. The key observation is that the image bytes are already live: there is a single bootc-owned containers-storage shared by all deployments, and quadlets read it via `additionalimagestore`. What is tied to a deployment is only the definition (the `bound-images.d` symlink and the quadlet it points to), which the running system reads from its own `/usr`. So no overmounting is needed: changed quadlets are written to `/run/containers/systemd/`, which podman gives precedence over `/etc` and `/usr`, systemd is reloaded, and the affected `.container` units restarted. Because `/run` is transient the override disappears on reboot, at which point the staged deployment's content applies and the system converges on its own. A soft reboot preserves `/run`, so the override is cleared when one is prepared. The command refuses to run unless the ostree diff between the booted and staged commits is entirely accounted for by the bound image changes it will apply, so it never silently applies half an upgrade; since the diff doesn't enumerate added/removed directories, those are walked in the deployment roots. The applied state is recorded in `/run/bootc/apply-live/bound-images.json` before any unit is touched, with each unit marked pending until its restart succeeds, so a failed restart (or `--no-restart`) is retried by simply re-running. The state is surfaced in `bootc status`, and its image names become extra GC roots so a `bootc rollback` that discards the staged deployment doesn't prune an image a running container uses. A later staged deployment reverts any overrides from a previous apply that it no longer needs. The verb is a subcommand group (`apply-live bound-images`, alias `lbi`) so that the bare `apply-live` stays reserved for the general feature and further scopes can be added without a new top-level verb. It is hidden and experimental, and ostree-only for now; composefs needs a tree diff API before the scope check can be implemented there. This has unit coverage for the planning, scope check and state logic. The /run write, SELinux labeling, systemctl sequencing, status output, GC root handling, out-of-scope refusal and post-reboot convergence were verified on a CentOS Stream 10 VM by the tmt test in the following commit. Assisted-by: AI
The unit tests cover the planning, scope check and state file logic, but not the parts that only exist on a real host: writing to `/run/containers/systemd` with the right SELinux label, the systemctl sequencing, and the interaction with `bootc status`, garbage collection and reboot. Switch to an image with a bound `.container` that just sleeps, then upgrade to one that only changes the bound image, and walk through `--dry-run`, `--no-restart` (unit still on the old image, state marked pending), a plain re-run (unit restarted onto the new image), and the idempotent no-op. Discarding the staged deployment via rollback and running cleanup checks the live state acts as a GC root. An upgrade with an unrelated file change checks the out-of-scope refusal. After the reboot, the `/run` override and state must be gone and the deployment's own definition in effect. Assisted-by: AI
Apply two pieces of feedback from cgwalters's review of bootc-dev#2458: - Scope the module doc's "doesn't require mounting anything" claim to the ostree backend, since the composefs backend lays out image storage differently. - Replace the `/run/bootc/apply-live/bound-images.json` state file with state embedded directly in the quadlet files written to `/run/containers/systemd/`: each one now carries a leading marker comment recording its staged checksum, image reference, and whether its unit still needs a restart. That directory is now the source of truth instead of a bespoke JSON file. A small flat-text list of protected image references is still written separately, since a bound image with an unchanged definition has no quadlet override to carry that state, and this list is only used to protect such images from garbage collection. Also address several correctness issues GitHub Copilot's automated review found in that state-tracking code: - Track the previous unit name across a `ServiceName=` change so the old unit is actually stopped instead of left running alongside the new one. - Merge newly staged protected images into the existing `protected-images.list` instead of replacing it, so an earlier deployment's still-running bound image isn't GC'd out from under it. - Make the fast path compare the full set of expected quadlets, not just checksums, so a retry after a partially-failed apply doesn't mistake incomplete state for complete. - Refuse to overwrite a `/run` quadlet override bootc didn't write itself. - Guard against an Added/Removed pair that target the same `/run` destination (e.g. a quadlet moving between search dirs) from deleting the replacement that was just written. - Always refresh a quadlet's marker checksum, even when its contents are unchanged, so status and the fast path stay accurate across later deployments. - Skip symlinks and other non-regular files when scanning `/run/containers/systemd/`, so a unit masked via `-> /dev/null` doesn't break `bootc status`. - Update the booted integration test to read state from `bootc status --json`'s `liveBoundImages` and `protected-images.list` instead of the now-removed JSON state file. Still under review; not yet addressing the separate comment about whether `liveBoundImages` belongs in `bootc status`/the schema, nor the deeper `/etc`-vs-vendor quadlet precedence question Copilot also raised. Assisted-by: AI
Reuse the apply-live fixes from jmarrero-forge#10 (0881235, 00b527a, b428ed4, 636c124). Cherry-pick conflicted against the marker-based tip, so those commit trees are adopted here: - Gate the fast path on applied-checksum; always run the scope check - Plan stops/restarts from live state (pending-restarts / stopped-units) - Refuse non-bootc /run overrides and /etc-shadowed vendor quadlets - Restart units an earlier live removal stopped when they return - Extend the booted tmt coverage for those cases Generated-by: AI
Preparing soft-reboot without --apply leaves containers running; clearing /run overrides and protected images then lets GC prune in-use images and drops definitions while units still use them. Clear only when --apply triggers the reboot handoff. Assisted-by: AI
Joseph asked whether live-applied bound images belong in HostStatus / the schema. Live state already lives in /run markers (/run/containers/systemd and /run/bootc/apply-live); surfacing it in status duplicates that and couples the schema to an experimental feature. Drop the types, status wiring, schema defs, and point the booted tmt test at the markers instead. Assisted-by: AI Signed-off-by: Yasmin de Souza <ydesouza@redhat.com>
Number 49 is already taken by test-49-composefs-1-16-bridge; the generated fmf already registered this test as 68. Align the # number comment so `cargo xtask update-generated direct --check` passes. Assisted-by: AI
Author
|
@jmarrero I am not sure why but I am unable to run the CI. Can you run it for me? |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Rebase and cleanup of the apply-live bound-images work from #2458, incorporating the follow-up fixes from jmarrero-forge#10 (fast path, unit reconciliation, override checks), plus review follow-ups: soft-reboot clear only at
--applyhandoff, and droppingliveBoundImagesfrom status/schema in favor of/runmarkers.Test plan
cargo test -p bootc-lib --libjust test-tmt logically-bound-apply-live