Repository navigation
store/s3: sparse-aware, chunked, content-addressed checkpoint transfer - #364
Merged
Merged
Conversation
An export file is now a manifest of 16 MiB chunks keyed by SHA-256. Upload walks the data segments with SEEK_DATA/SEEK_HOLE, skips holes and all-zero chunks, and hashes each chunk once for both its key and the record digest. Fetch sizes the file and writes only the present chunks, so a mostly-hole cow.raw stays sparse on the fetching node and no longer travels at its logical size. The record digest is unchanged for the same content. Records written in the old layout are not readable.
Chunks live under chunks/<lineage>/<sha256>. A publish lists its lineage once and uploads only chunks the lineage lacks; the lineage defaults to the record id and a claim's captures pass the sandbox id so consecutive snapshots share their unchanged chunks. Delete removes a record's manifests and meta. SweepGenerations, a no-op until now, drops superseded manifests idle past the grace and then chunks that no manifest names and that are older than the grace, so an in-flight publish's fresh chunks survive. Publish gains a PublishOption and SweepGenerations a context; the dir backend ignores the option.
A record pulled for peer heal no longer streams holes. Each export file is a size entry followed by one tar entry per non-zero chunk carrying its index and SHA-256; the receiver sizes the file, writes the chunks at their offsets and fails the pull on a checksum mismatch. Endpoint and auth are unchanged.
Mutation checks showed the sparse tests passing without a trailing hole, a repeated extent in one chunk or a young manifest beside an old meta. Each now has a case that fails when its guard is removed.
A publish used to trust the lineage chunk listing, so it could skip uploading an unreferenced chunk that a concurrent sweep was about to delete. It now reads the committed manifests of the lineage and reuses only the chunks they name; any other chunk is uploaded again. Accepted residual: a sweeper that listed an old unreferenced chunk can still delete the fresh re-upload of the same key between its list and its delete, which needs a content repeat and a concurrent sweep.
A record deleted between a publish's reference read and its manifest upload used to lose the chunks the new record dedups. Delete now removes only meta.json and the labels and re-stamps the generation manifests; the sweep counts a manifest without a meta as a reference until it has been idle for the grace and frees its chunks only afterwards. A fetch that already selected the record also survives its deletion.
A re-publish between the sweep's listing and its meta GET left the listing's stale mtime in charge, so the sweep took the new generation for an old one and deleted chunks it dedups. The age now comes from the GET response that also names the current generation.
A publish into a shared lineage listed every record of the namespace and read two objects per record. It now writes a zero-byte marker lineages/<lineage>/<id> before the commit and reads only the records the markers name: one LIST plus two GETs per record of the lineage. The sweep removes a marker once its record has neither meta nor manifests and the marker is idle for the grace.
The checkpoint and template views of an s3 store share one prefix and the s3 sweep already reads both id namespaces, so sweeping the template view repeated the whole listing and widened every race window. The template view is skipped when the checkpoint store is shared.
Publish and PublishDigested take a lineage string instead of a functional-options set. s3.go places its type blocks directly under the Store methods and every utility after them, and no local shadows the path package. ScanFile reads with ReadAt, the zero-length manifest slice literal and a wrong comment tail are gone, and the WriteChunk cases carry an index field instead of branching on the case name.
When a record's meta disappeared between the sweep's listing and its meta GET, the sweep skipped the record's manifests, so a chunk another record of the lineage had just reused from them was deleted. The listed manifests now still count as references and are not removed, leaving reclamation to a later sweep.
Dedup safety relies on a publish finishing within store.GenerationGrace after the record it reused chunks from is soft-deleted. Without a bound, a slow publish could commit a manifest naming a chunk the sweep already deleted, and the hole would spread to every later capture through the manifests the dedup set is built from. Run publish under the budget fetch already uses, renamed transferBudget: a record that cannot be uploaded in that budget could not be fetched within it either, and the budget stays below the grace so the reuse window always covers a publish.
… Delete Delete removes only meta.json and the labels, so a soft-deleted record kept its <prefix><id>/ prefix (its manifests) for the grace plus up to one sweep period. The delimiter listing returned every such prefix and Metas paid a 404 meta GET for each; with replication publishing a checkpoint every interval and soft-deleting the previous one, GET /v1/checkpoints, the TTL sweep and restart reclaim scaled with (grace / interval) x replicated claims. Manifests move to <prefix>manifests/<id>/<gen>.json, beside chunks/ and lineages/, so the record prefix exists exactly while the record is live and the id pattern never matches the manifests/ prefix. Sweep semantics are unchanged: manifests of a live record are references, manifests of a vanished record stay references until idle past the grace and are then doomed, and chunks are freed only when no non-doomed manifest names them. Delete also removes the record's lineage markers at once, reading the lineage from the manifests it already refreshes. Markers only feed the dedup set, which resolves each marker through the record's meta, and the sweep's marker reclaim; chunk references come from manifests alone. The sweep's stale-marker pass stays for crash residue. The first capture after a wake uploads every chunk again because the wake deletes the record it restored from; deploy.md says so.
sandboxd no longer garbage-collects the S3 store: SweepGenerations is a no-op again and the sweep, its inventory and the stale-marker pass are gone. Delete keeps its soft semantics (drop meta, labels and the lineage markers, re-stamp the manifests so their LastModified is the delete time), which is what makes an external reclaim safe against in-flight publishes together with the 30 minute publish bound. deploy.md documents the key layout, what sandboxd never deletes, the safe reclaim rule, the invariant behind it and the cost of not running one.
This was referenced Oct 18, 2026
This was referenced Oct 18, 2026
This was referenced Oct 18, 2026
CMGS
added a commit
that referenced
this pull request
Nov 4, 2026
* silkd: a client hang-up reaps an attached language server The pump only watched the server's stdout, so a silent server that ignored stdin EOF pinned its memory after the client left. A request frame during the stream ends it the same way, as a pty does. * silkd: git_branch list names an unborn branch rev-parse --abbrev-ref fails on an unborn HEAD and printed HEAD, while git_status on the same repo reports the branch. A detached HEAD still reads HEAD. * review: silkd docs: attach never ends in done, a dropped port_forward can linger * review: silkd and sandbox-init borrow what they only read claim_env keeps its one read buffer and parses into borrowed pairs, so an exec, pty or session drops 2N String allocations. BootCfg and IpParam borrow the cmdline, read_manifest returns its text, apply_config sets literals, pty and upload signatures take plain borrows, and scan_serials formats a path only when it reads it. Tests changed only where a literal's type or a temporary's lifetime named the old owned form. * review: silkd drops code that cannot run Child::id is Some right after spawn, a session body is never empty, Path::file_name is never empty, abort on a finished task is a no-op, and the exec rebinds did nothing. Table::new and Broker::new only aliased Default; git hand-rolled error_frame; the imds consts and handle, DEFAULT_INIT and PROTO_VERSION had no outside user; the ignored bench asserted nothing. * review: silkd statics and test helpers follow the file layout Function-local statics and consts move to the top-of-file block, and the one-caller proxy_vars and silkd_net wrappers go with them; test helpers move above the tests and private below public. * review: silkd comments shrink to what the code cannot carry Module narratives go to one line, the imds token fact moves to its const, a history comment goes, make_controlling_tty becomes safe because it has no memory precondition, and find's at_root stops shadowing the root param. * review: silkd tests use the shared request and assert_error helpers * sdk/python: ship the PEP 561 py.typed marker The package is fully annotated and mypy-strict, but the wheel carried no marker, so a consumer's type checker treated every cocoonsandbox import as untyped (reveal_type gave Any). Add the marker and the setuptools package-data entry; a scratch wheel now lists cocoonsandbox/py.typed and mypy on a consumer reveals cocoonsandbox.sandbox.Sandbox. * review: one-line docstrings where several lines restated the signature or sibling docs * review: drop duplicate test fixtures and a subset test; share FakeConn through conftest * review: tests before helpers in test_stream and test_client * review: adapters floor the SDK at 0.1.13, where the in-guest kill after a timeout cut first shipped * review: silkdtest's default arm calls serveStateless The arm re-spelled serveStateless byte for byte. * review: trim multi-line comments on unexported decls, capitalise own-line comments Own-line comments start uppercase per /code Comment Style; multi-line godocs on unexported names cut to one line. * review: guestserver keeps its report utility below the halfCloseSink block Layout only: /code File Organization puts utility funcs below the type's methods. * review: volumesmoke rejects non-local probes with filepath.IsLocal Same verdict as the hand-rolled path test on 15 inputs. * review: harness owns the guest proxy address, RelayConn and a package-level Request The proxy address was declared in four drivers. Dial returns RelayConn so portsmoke calls CloseWrite directly. PortRelay.Request never read its receiver. credsmoke's expectPlaceholder locals stop shadowing the entry and header consts. * docs: deploy's upgrade notes name the release each starts from At v0.1.21 "this release" read as every upgrade needing a drain, a fresh data dir and an emptied bucket. Releases come from the first tag holding each commit: #81 v0.1.6 (the pool key's engine axis went), #364 v0.1.18, #59 v0.1.5, #69 v0.1.5 and #72 v0.1.6, #171 and #169 v0.1.11. * docs: deploy and egress samples load and name a pulled image by its full ref A pulled image is keyed by its whole ref, so template base:24.04 fails the golden build with image not found, as the README already fixed in #425. The deploy sample's cluster_key and the egress sample's missing egress_ca failed config.Load. * docs: deploy states what the code requires of a failed claim, a golden and a dump A wake of a failed egress-lane or volume claim answers 409, tenants get 403 on two more root routes, golden dumps queue, an idle default acts on running claims at the next sweep, and an adopted replica always wakes on a read; none of these had a line. * docs: deploy gives the archive registry's DDL A role without CREATE could not enable archive_failover from the docs: the other three shared tables have a pre-create DDL and this one had none. * docs: deploy keeps the S3 layout and the replica bound in one place each The checkpoint_store row repeated the Reclaiming S3 storage section, which takes the facts only the row carried (lineage scope, marker order, re-stamped manifests); the TTL, replica-loss, tenant-outage and cluster-invariant passages become links to the pages that own them. * docs: cluster lists registered volumes and the TTL, and says when the digest covers secrets An operator rebuilding data_dir from config.json lost runtime-volumes.json silently, and gauge 0 read as a check of two secrets a keyless digest never covers. * docs: security names the TLS edge, the re-admitted prefix and the shared usage table Stated as hard constraints these ruled out what deploy ships: a TLS edge for SDK clients, egress_internal_allow, and meta_store.usage. * docs: benchmarks and performance keep each caveat once * docs: api states redirects, operator tokens, fork env and the checkpoint list as the server does The mint endpoint never answers the 503 and 429 its paragraph listed: the browser request does. A template another node deleted stays in tplSet, so the cluster page's never-fires claim was false for that entry. * docs: api lists the errors of a call that reaches a guest once Wake, agent, ports and exec carried the same four tails. Restore failures, 409s, release, hibernate and the preview listener differ, so they keep their own lines. * docs: sdk, silkd, langchain and android match what the code does Go Exec runs the argv given, with no implicit shell; every control-plane verb returns APIError; the checkpoint option refusals and a fork child's env were incomplete. * docs: the sdk pages link the volume and deadline rules the api page owns Go-specific and Python-specific names stay. The settled-entry sentence stays on the Go page, its only source. * sandboxd: a preview listener that cannot bind is fatal A taken preview_listen was logged and ignored: the node stayed healthy while the preview endpoint kept minting URLs for a dead port. The main listener already exits on the same failure, and systemd Restart= is the retry loop. http.ErrServerClosed stays clean. * tenants: the contract replaces the set with no tenants An empty Replace must delete every tenant on both backends: pgx sends a nil slice as NULL, so a Collect-built name list would delete nothing on pg. * review: one owner for the max fork default, the routable-host test, the s3 store test and the tenant claim mapping config.DefaultMaxForkCount replaces the second literal in pool, config.RoutableHost replaces the copy in server.New, Config.S3Checkpoints replaces three spellings of the kind test, and forTenant replaces the tenant-to-options mapping written in both claim paths. Behaviour-preserving, net zero lines. * review: store package docs fit the comment budget, constant errors use errors.New Package docs of store, s3 and peer shrink to one line (docs/deploy.md owns the key layout); the SweepStaging godoc and the redirectCacheEntry comment restate their names; three message-only fmt.Errorf become errors.New; VolumePatch is decode-only, so its slices follow the omitempty rule. * pool: a fenced node captures no replicated claim Fork, Checkpoint and Promote reached captureAllowed with no fence gate, so a node past its lease window could hand out children, a checkpoint or a template from a claim a peer may already own. ErrLeaseLost maps to 503 through the error table and is benign to the archive and replicate sweeps. * pool: consumeArchive reads the row epoch under the manager lock An unfence rewrites ArchiveEpoch holding only m.mu, and a release or replica retirement read it holding only Transition, so the two raced. The stale read cost at most a refused Consume and a settle round trip. * review: pool comments name what the code does now Two field comments named fields that moved to configView, checkTemplateOwner never deletes, HasPromotedTemplate's cache can go stale, and poolSeedHash's field list drifted against the lines below it. * review: checkpoint.go keeps const and var above its types * review: fold pool helpers with one caller archiveDeleteFor, unpooledTemplates, archiveFetchErr, dropStaleRow, hasRegularFile, deliverGuestEnv and TenantRecords each had one caller. reputArchive's held parameter was dead: its only caller guards a non-zero epoch. * review: one owner for pool facts stated twice replicaPools repeated view.poolReplicate, whose key set a reload cannot change and PUT /v1/pools cannot set. The view's map guards bought nothing, since every reader takes the zero value. env and replicable spelled out runningShaped, AgentSocket spelled out isFenced, and the two reader teardowns stripped a cancel that removeVM strips itself. addClaimLocked gains one atomic load on nodes with archives and nothing elsewhere. * review: pool sweeps and wakes carry what they have commitWake logged its persist failure at Warn and its caller rebuilt the error with the cause dropped; it now returns the error. idleOnce copied id and token so idleHibernate could look the claim up again, and hibernateLocked's setIfLive already answers ErrUnknownSandbox for a released claim. Fork and Checkpoint no longer wait on Transition for a usage field, since every write of VMName after publication holds m.mu; lockedVMName stays for tests, which read under Transition. * review: types.go splits into env.go and volume.go Pure move: the env types, limits and patterns go to env.go and the volume types, modes and mount validation to volume.go, matching the existing env_test.go and volume_test.go. envView and namedVar stay beside the Sandbox that owns them. No line changes beyond imports and gofmt alignment. * egress: an upstream dying mid-body truncates the guest's response relay dropped the copy error, so net/http wrote the terminating chunk and the guest read a partial body as complete. Hijack and close the conn on a copy error, which leaves the chunked stream unterminated. Success costs one nil check. * outbound: ArmProxy closes a displaced door before it binds The close unlinks the socket path, so a re-arm that bound first left the guest with no door while Route reported relay. A re-arm with no policy left now closes the door too, which makes the Disarm in restartLocked dead. The claim path pays one map delete under a lock it already holds. * engine: a restore gets the dump budget A dump may take 15 minutes while an eager-copy restore of the same image had 2, so a slow page cache could fail the wake it had just made possible. cocoon gives vm.restore the same limit as vm.snapshot. Clone and cold boot keep the command budget. * review: drop checks and fields the code already guarantees ReadSlice caps a line at the reader's buffer, IsGlobalUnicast already excludes link-local and multicast, UnixListener.Close already unlinks, marshalCA's public key is its PEM key's, and interDER copied interCert.Raw. * review: DialFunc's Router methods serve only tests The only production egress.New takes a claim as its router, so the two methods move to the egress test file. * review: a CA key error names the key its caller parsed parseECKey said intermediate key while parsing the root, so IssueIntermediate reported root key: intermediate key. validHostName takes the package's maxHostLen for the same DNS limit. * review: cut a restating godoc and fold the engine package doc to one line * review: layout: silkd helpers live in silkd.go, tcpAddrPort beside proxy.go's other address helpers installca.go kept the generic guest write and exec helpers every other engine file uses; installOutputMax becomes guestOutputMax since it caps every guest exec. * docs: name the capture fence, the restore budget, a preview bind failure and a cut upstream * review: VolumePatch keeps the SDK's omitzero tags, as the wire parity test requires * engine: a clone restores memory like a restore, so it gets the dump budget cocoon's vm clone and vm restore share restoreVM and its 10 minute memory transfer, and an archived wake and a fork go through clone. A hung refill clone now holds its slot for up to 18 minutes, which stays bounded and loud. * review: silkd's language server feed needs no flag or channel, and one helper sets the claim env * review: refill.go keeps the refill loop; golden builds go to golden.go, VM provisioning to provision.go, destroyAll and dropSnap beside destroy, runBounded to pool.go with its callers across the package * review: the reaper leaves claim.go for reap.go, since expiry handling is not claim admission * review: claim and authed resolve a credential, so they sit in operator.go with resolve and byID * review: the pool type joins its watermark methods in watermark.go, so its type block is atomic * review: rewriteArchiveRow writes an archive row, so it sits in failover.go beside refusedReplace * review: NewManager delegates template and volume seeding to loadTemplates in template.go and seedVolumes in volume_catalog.go, keeping the order * review: the catalog owns its entries and holders, so catalogVolume, catalogSnapshot and the holder bookkeeping move to volume_catalog.go as volumeCatalog methods * review: the seven single-flight sweep guards group into one sweeps field on Manager * review: recordLocks owns the per-record lock maps and their mutex, and recLock's doc says it returns the lock without taking it * review: healVetoes owns the heal veto maps and their mutex, replacing the three Manager methods and fields * review: ckMarkers owns the archive delete markers, and the delete and retry chain moves with it to archive_delete.go * pool: an uncordon during an evacuation's drain stops it Evacuate waits for the warm VMs to go before it starts its run. A DELETE /v1/drain in that wait found nothing to stop, and the run then moved every claim off an uncordoned node. * pool: a tier claim woken past its lease before the export gets a fresh lease The reaper archives a hibernated claim of a tier pool at its lease end, and the same call a moment after the export restores it with a fresh lease. A call in between woke it with the past deadline, so the next reap tick destroyed it. * pool: a running claim that moved to a peer closes its egress doors forgetAdopted removes the VM with no sandbox id, so the removal never disarms. The doors, their listeners and their goroutines stayed until the next restart, one set per moved claim, unlogged. * docs: a moved claim's token, its first wake, and the 503s an evacuation and the tenant store answer * sdk: Close follows a sandbox that moved off a retired node Close looked the sandbox up only on a 404 from its old owner. Once that node is retired, the first release fails to connect, so the claim stayed on the adopter until its deadline, and every retry hit the same dead address. A connection failure now looks it up too, and keeps its error when no node answers.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #362. Refs #183 (sandboxd no longer reclaims S3 at all; the reclaim contract is documented for the operator instead, see below).
Bottom of a stack: #366 builds on this branch, and #365 on #366. Merge in that order.
What
An S3 export file is now a manifest of fixed 16 MiB chunks (the existing
DigestChunkSize), each stored under its SHA-256.<prefix><id>/meta.json(andlabels.json) is the live record.meta.jsonis uploaded last as the commit marker, and the record prefix exists only while the record is live.<prefix>manifests/<id>/<generation>.jsonlists each file's path, size and present chunks, plus the lineage.<prefix>chunks/<lineage>/<sha256>.<prefix>lineages/<lineage>/<id>.ScanFilewalks data segments withSEEK_DATA/SEEK_HOLE, so a chunk lying fully in a hole is never read, and a chunk read as all zero is skipped. The chunk SHA-256 is both the object key and the record-digest chunk entry, so there is no second hash pass; the record digest (PublishDigested) is unchanged for the same content. A publish runs under a 30-minute budget, the same as a fetch.ErrNotFound.Publish/PublishDigestedtake a lineage (""means the record id). The pool passes the sandbox id, so every checkpoint, archive and replica of one claim shares chunks. A publish reads the current manifests of the records its lineage markers list and uploads only the chunks they lack.meta.json, the labels and the lineage markers, and re-stamps the record's manifests. It never deletes a chunk or a manifest, so a fetch that already selected the record still completes.SweepGenerationsstays a no-op, as on main. Reclaiming chunks and manifests is an offline operator job:docs/deploy.md→ Reclaiming S3 storage documents the key layout and the procedure. Stop every node on the prefix, build one inventory, keep the manifests live metas select, delete the rest and unreferenced chunks. Online reclamation is not supported. The dir backend keeps its own local generation sweep.transfermanageris gone: chunks are at most one part (go.mod−1).Hot-path cost
None on the claim path or the data plane.
Metas(checkpoint listing): one GET per live record. A deleted record leaves no prefix to list.Owner decision: no compatibility, upgrade step
Records in the old S3 layout are not readable after this change, and there is no fallback reader. Before starting the new binary on an S3
checkpoint_store, do one of these:prefixat an empty location.docs/deploy.mddescribes this and warns against an expiry lifecycle rule on the prefix, which would delete chunks a live record still names.Tests
Metassends no meta GET for it, and its chunks survive.SweepGenerationsleaves the bucket untouched.Hardware acceptance
Two bare-metal nodes (one cell, PostgreSQL registry, S3 via SeaweedFS 4.48, cocoon v0.7.3,
rt:24.04, 10 GiB disk). The stack head contains this branch's final code; later commits on this branch are docs only.meta.json, alabels.jsonor a lineage marker, with 0 underchunks/ormanifests/. All 1,739 chunk and manifest keys ever seen were still present at the end.GET /v1/checkpointsstayed at 0.15–0.20 s.Review