Repository navigation
image: Store filesystem snapshots in manifests - #406
Open
daandemeyer wants to merge 7 commits into
Open
daandemeyer wants to merge 7 commits into
daandemeyer wants to merge 7 commits into
Conversation
The manifest uses the wrong timestamp field and lacks the trailer that marks a complete sequence. Let's use mTimeNSec and require the trailer in readers and writers. Let's also move the format out of the image driver into a uapi16 package of its own. Other packages can then read and write manifests without depending on image/. Signed-off-by: Daan De Meyer <daan@amutable.com>
Buck directory artifacts lose inode metadata and hardlink identity. Let's keep complete filesystem manifests beside new content blobs and restore private roots before running commands, with declared hardlinks in the upper so index=off can't split them on copy-up. Signed-off-by: Daan De Meyer <daan@amutable.com>
Image layers lose package and operation metadata when Buck stores their directory artifacts. Let's capture snapshots in the producing action and restore them for image commands, solvers and terminals while keeping box layers on their direct mounting path. Signed-off-by: Daan De Meyer <daan@amutable.com>
Terminal archives and directory artifacts can lose metadata or mistake shared contents for shared inodes. Let's write archives from snapshots, keep terminal listings consistent with their outputs, and restore directory exports only after Buck has materialized their manifests and blobs. Signed-off-by: Daan De Meyer <daan@amutable.com>
Image reconstruction copies every regular file before commands run. Let's use private reflinks when available and metadata over shared blobs otherwise. Declared hardlink groups share private inodes. Capture uses upper inode identity when OverlayFS reports different inode numbers for paths linked by commands. Signed-off-by: Daan De Meyer <daan@amutable.com>
Box layers lose modes, hardlinks and xattrs when Buck stores them, so a child box installs over a parent that differs from what the package manager wrote. Let's capture every box layer as a snapshot and install a child box over its restored parent snapshot. The sandbox keeps mounting one plain directory per layer, which we export from the snapshot with only what Buck stores, so action startup doesn't change. Signed-off-by: Daan De Meyer <daan@amutable.com>
Buildroots and their BuildRequires deltas are still overlay uppers with whiteout and opaque markers encoded as files, while boxes and images use snapshots. Let's capture every install as a snapshot and restore it for plans and builds. rootfs then only needs bind mounts, and we can drop its marker translation. Every lower root of an install is now a restored snapshot, so the package installers no longer need to know whether they install over lower layers. Signed-off-by: Daan De Meyer <daan@amutable.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Buck directory artifacts lose inode metadata and cannot preserve every logical
filename. Let's capture complete UAPI.16 snapshots in image-producing actions
and store regular contents in immutable SHA-256 blobs. Derived images retain
parent stores as explicit inputs and add only newly introduced blobs. Metadata
changes affect cache keys without duplicating unchanged data. UAPI.16 records
use mTimeNSec and the required trailer.
Image roots first try explicit reflinks into temporary private files. Independent
files have separate inodes; declared hardlink groups share only their private
inode. Unsupported or denied reflinks discard the partial tree and select an
OverlayFS root with separate lower metadata and data-only blob layers. Writable
hardlink groups start as private upper files. Selection uses actual clone support,
so separate btrfs filesystems do not accidentally trigger full byte copies.
Read-only roots use either a read-only bind mount or a read-only overlay.
Unprivileged OverlayFS can report different inode numbers when a command adds a
hardlink to an inherited file. Capture uses the actual upper inode to preserve
that group in snapshots and exports. Repart and terminal readers that apply
tmpfiles use full private roots when their consumers need consistent raw inode
identity. Metadata restoration and later writes leave cached blobs unchanged.
Boxes retain their directory layers and direct startup path.
Archive writers consume manifests and blobs directly when possible. Terminal
listings describe the processed output, including package-database removal and
timestamp clamping. Directory outputs keep an authoritative manifest and provide
an exporter that restores metadata after Buck materialization.
Exports require a new destination under a parent owned by the caller or root,
with sticky protection when group or other write is allowed. Caller-owned
parents also work when Buck runs inside a user namespace.
Directory export needs read access to the kernel's UID overflow sysctl to
distinguish caller ownership from unmapped users.
On a same-filesystem btrfs input with 4,097 paths and 64 MiB of logical contents,
three-run median writable setup takes 162 ms with private reflinks and 139 ms
with a forced overlay. Chmod of 100 files takes 0.42 ms instead of 2.42 ms, and
100 partial writes take 13 ms instead of 20 ms. Read-only setup takes about
157 ms with either strategy. FIEMAP confirms shared extents for the private
files. Earlier non-reflinking tmpfs measurements cover memory copies rather
than ext4 disk I/O.
The native example digests change for the initrd, UKI, disk and three manifests.
The kernel, /usr filesystem, verity data, root hash and extension are unchanged.
Arm64 reference digests still require a native refresh. Device-node export
requires a filesystem and namespace that permit mknod; cpio retains whole-second
timestamps and cannot represent user extended attributes.