Repository navigation
fix: close device, stream, and preview lifecycle gaps - #81
Merged
Merged
Conversation
CMGS
marked this pull request as ready for review
August 13, 2026 18:16
This was referenced Aug 13, 2026
The overflow flag was only checked at the loop top, so a callback preempted between a full try_send and the store could leave the loop parked on an empty channel forever - a silent quiet watch, the exact loss the terminal error exists to prevent. Dropping the sender makes the closed channel the wakeup itself: the buffered ordered prefix drains first, then recv() returns None and the terminal overflow error goes out. Deletes the Arc/atomic machinery outright. The pty close error never reached readers (pipe readers always see ErrClosedPipe), so plain Close replaces CloseWithError.
DisableKeepAlives paid a measured +0.23ms relay dial per request to get per-request revocation, activity, and audit. PreviewTouch does the same three against the claim map for microseconds, so the kept-alive pool returns for sub-resource fan-out; the idle pool stays keyed per claim. Measured on .79: preview seq p50 back to the pre-change band (0.17-0.21ms vs 0.44ms), warm claims unchanged. deploy.md documents the bearer-token payload and the shared browser origin under one preview_advertise.
Reattach the preview_advertise doc to its field at one line, drop two edit-narration clauses and a restating test comment, compress the watch overflow WHYs, and make PortConn.Close match Pty.Close (the writer-side error value never reaches readers).
resolveGolden decoded the template record and dropped it, so a tenant token could claim any other tenant's promoted template by name and boot its snapshot - proven with a test that saw the owner's content digest come back under a foreign tenant. Promote and delete were already tenant-scoped; only the read path was not. A foreign template now resolves as absent, which is byte-identical to a name that was never promoted, so existence stays unconfirmed. Root-promoted templates remain shared, as pools are.
forget dropped a dead peer from the view, but merge re-inserted any node it did not already know, so one push/pull from a peer that had not yet seen the death resurrected it - permanently, since SWIM never fires NotifyLeave twice, and the node was then re-gossiped cluster-wide. Tombstone the epoch it left at: a lagging peer replaying that epoch is ignored, while the node's own restart always seeds a higher epoch and clears the tombstone.
The audit op is preview, once per request, not preview_dial. pty_open, lsp_request and port_forward are connection-bound alongside fs_watch. Volume discovery returns the gossiped union only to root; a tenant sees this node's local entries its ACL permits.
The Forget-mid-flight test spun on an atomic with no deadline, so a probe that never arrived hung until the binary timeout instead of failing; wait on a channel the handler closes. The swallows-failures test never checked the healthy peer was reached, so it passed even if the fan-out stopped at the first error - count that peer's hits.
A tenant configured without an egress block inherited the pool's entire allow-list including its secret injections, so adding a tenant token handed that tenant the operator's credentials - the opposite of the documented intersection. Both single-sided branches now deny: a tenant reaches only what both layers allow, and a pool without a policy grants nothing to anyone. Root keeps the pool policy whole, having no tenant layer to intersect.
Document engine as the fourth pool-key axis (claim body, pools table, and the three key examples that omitted it), list the archived/draining/digest-mismatch gauges and the archive counters /metrics actually emits, add the secrets and egress_internal_allow rows, name the 401 the keyed HEAD probe answers, and say the usage journal's key is the pool key's hash. Also: the audit record's egress decision/secret fields, sandboxd.dbg in the release assets, browser is amd64-only while base is multi-arch, browser in the os-image list, bench.sh and release.yml in the repo map, and the nft lock drops guest-initiated packets. Drops a third-party registry namespace from the android README.
Guest connection pooling let a reused preview request skip PreviewDial, and with it wakeResolved and the Transition lock that serializes a claim against its own hibernation: once the idle sweep passed its re-check and entered the seconds-long snapshot, a pooled request wrote into a freezing VM and never triggered the wake. Every request dials again, so authorization, the activity stamp, the audit record and the wake all ride the one path. Egress pools accepted idle_hibernate_seconds even though the lane refuses to hibernate at all, so the setting could only ever produce a failure. Config now rejects it, and the sweep skips egress claims of unpooled keys, which the config gate cannot reach - futile work whose only symptom was a logged error every tick, so nothing observable is left to assert.
CMGS
force-pushed
the
codex/close-repo-review
branch
from
August 13, 2026 19:29
4183bb3 to
a951bf8
Compare
The epoch tombstone the previous commit added carried two residuals of its own: it lived only in memory, so a restart forgot every death and a lagging peer could teach them back, and its entries were dropped only when the same node returned with a higher epoch, so a permanently retired node_id stayed forever. Membership already answers the question - merge now accepts gossip only about nodes SWIM currently reports, and NotifyJoin/NotifyLeave maintain that set. Both maps are bounded by live membership and rebuilt from SWIM on restart, and a node returns to the view by rejoining, not by out-numbering a tombstone.
Every base image was a floating tag, so the kernel build, sandbox-init and silkd were reproducible only by luck, and rust-toolchain named 'stable' — a new release could change codegen or fail -D warnings with nothing changed here. Bases are pinned by digest and both toolchains to the version the Linux gate actually runs. A pinned digest goes stale silently, so dependabot owns refreshing them (plus cargo, gomod and actions) rather than a human remembering. shellcheck had no gate at all: all seven scripts already pass, so this locks in what is already true.
httpError becomes APIError: the SDK's own redirect walk branches on the status, so a caller distinguishing gone from full from mid-heal needed the same field Python already exposes. WithHTTPClient lets a caller supply its own transport or deadline; the client sets no blanket one, because checkpoint and promote block for as long as the snapshot takes and ctx is where a Go caller says otherwise. claim_ref had no SDK at all despite being the aggregated apiserver's hook, and neither did the listing that reads it back, so both sides gain WithClaimRef / claim_ref and Sandboxes() / sandboxes(). Python gains attach, drain and uncordon: the OpenAI adapter was reaching around the missing attach by constructing a Sandbox by hand. MCP claimed with the node's 5-minute default and renewed nothing, so an agent session outliving it lost every sandbox mid-conversation; it now claims for an hour, says so in the tool description, accepts net and size, and releases what it claimed when the stdio session ends.
…mantics cluster.md told operators to call SetPools and Drain, which docs/sdk.md never mentioned despite claiming a per-method reference; both SDK pages now carry them alongside the new claim_ref and sandboxes surfaces. The checkpoint-delete lifecycle was written out in full in three places, so the API page keeps what a caller of that endpoint needs and links cluster.md for the rest.
exec() threw stdout away on a non-zero exit, so a failing build's log was lost where Go returns it alongside ExitError. Pty dropped the shell's exit code the exit frame carries, and Watcher ended iteration identically for a clean close and a dropped relay, so a caller could not tell them apart. Go gains Checkpoint(id), which is what Python already had and what MCP needed — it was listing every checkpoint on the node to resolve one id, and only ever saw the connected node's listing. The two adapters floored the base SDK at 0.1 while calling APIs that landed in 0.1.5.
ExitError now carries the stdout produced before the failure, Pty exposes the shell's exit code, and a Watcher tells a dropped relay from a clean close.
…he docs This PR made engine a documented pool-key axis, which exposed two places that never had it. The SDK's PoolKey and PoolSpec omitted it, so an Info-then- SetPools round trip — a declarative full replace — silently drained every fc pool. The two pool gauges omitted it too, so two pools differing only by engine emitted the same label set and the scrape failed. The rest is truthfulness: resolveGolden's godoc still claimed only a true absence cold-boots, which the tenant gate changed; watch.rs still called itself the one connection-bound verb the docs page just stopped saying; the SetPools cluster example did not compile; ClaimRef claimed to be empty on warm-pool claims that in fact carry it; and the egress intersection change needed to say out loud that a tenant inheriting its pool's policy now reaches nothing. PortConn keeps CloseWithError(net.ErrClosed): it is a net.Conn, and callers test for that sentinel — only Pty, which is not, takes the plain Close.
The new shellcheck gate failed on its first CI run: SC2015 flags the cleanup idiom because C runs whether or not B did, which here is the intent but reads as if-then-else. Older shellcheck releases report it and the version this laptop has does not, so the code says what it means instead.
The drift guard had no assertion for either, and engine is exactly the axis whose absence from the SDK key silently drained fc pools on an Info-SetPools round trip. Mutating the tag to json:"-" fails this test.
CMGS
force-pushed
the
codex/close-repo-review
branch
from
August 13, 2026 20:25
977a3bf to
64c1aff
Compare
HasPromotedTemplate was tenant-blind while resolveGolden is not, so the two disagreed for a foreign tenant: routing saw a local golden, skipped the peer hop, and the claim cold-booted with routing believing a template had served it. Both now apply the same tenant test. The contract, stated precisely: a plain claim of a foreign name behaves exactly like a claim of a never-promoted name - the template axis is an image ref, so it cold-boots the public image or fails provisioning, and the owner's content is never touched. Answering 404 instead would be worse on two counts: it hands every tenant an existence oracle over other tenants' template names, and it lets one tenant deny everyone else a public image name by promoting a private template under it. Only require_promoted (the volume path, redirect retries) answers ErrUnknownTemplate. tplSet caches the owning tenant next to each id, so the ownership test stays in memory: the claim path must not grow an s3 read per warm miss.
create_sandbox claims for an hour, but fork passed a zero TTL and a branch claim set no timeout, so both fell back to the node's 5-minute default while being session-tracked with nothing to renew them - the exact mid-conversation expiry the create fix closed. Both now use the same lease and their tool descriptions say so. Also drops deploy.md's stale CH-only tag: Firecracker is a live opt-in engine axis, and the sentence's real content is that old state is not converted.
CMGS
force-pushed
the
codex/close-repo-review
branch
from
August 14, 2026 02:55
64c1aff to
5d8b083
Compare
The fc opt-in never carried its weight: it offered nothing CH does not (volumes, checkpoint, fork, hibernate and the restore modes are all CH; clone performance measured equal), it had zero e2e or hardware coverage, and the axis never made it through the template lifecycle - DELETE built its key without engine, so an fc-promoted template could never be deleted by name, and no SDK could send or read the axis at all. Rather than thread a one-value-in-practice axis through claim, promote, handles and delete, remove it: PoolKey is (template, net, size) again, the hash drops the engine component, the volumes-require-ch guard and the fc test rows go with it, and the pool gauges return to three labels. cocoon keeps --fc for anyone driving it directly. Old pool-key hashes change; upgrades already never convert state.
TemplateOwners answered from raw key hashes, so a foreign tenant's promoted template escalated another tenant's claim: a volume claim that would have cold-booted locally was forced onto the promoted path and refused, and a plain claim earned a useless redirect that also signalled the name exists. The local check is tenant-aware; the remote one had nothing to be aware with — gossip deliberately carries no tenant. Scope the hash instead: gossip advertises hash(keyHash|owner), and an owner query probes the requester's own scope plus the operator's (root probes every configured tenant). A foreign template simply never matches, which is exactly how the claim path treats it, and the wire still carries nothing but opaque hashes. Salting costs two sha256 on the warm-miss redirect path only; mixed-version meshes disagree on hashes, and upgrades are lockstep.
CMGS
force-pushed
the
codex/close-repo-review
branch
from
August 14, 2026 04:10
9c8b31d to
89304e5
Compare
This was referenced Aug 16, 2026
CMGS
added a commit
that referenced
this pull request
Aug 17, 2026
#81 made the effective policy a strict pool ∩ tenant intersection, and a key with no pool policy denies. A promoted template never has a pool — Promote and DeleteTemplate refuse pooled keys, and refill would build a golden for a name that is no image — so every promoted-template claim became default-deny regardless of the tenant's allow-list, closing the none lane's only route out for on-demand claims. Treat the missing pool layer as absent rather than empty: with no pool configured for the key, a tenant claim takes its own policy whole; a configured pool without a policy still grants nothing, and root claims of promoted templates stay denied for want of any layer. Closes #98.
CMGS
added a commit
that referenced
this pull request
Nov 4, 2026
* silkd: a client hang-up reaps an attached language server The pump only watched the server's stdout, so a silent server that ignored stdin EOF pinned its memory after the client left. A request frame during the stream ends it the same way, as a pty does. * silkd: git_branch list names an unborn branch rev-parse --abbrev-ref fails on an unborn HEAD and printed HEAD, while git_status on the same repo reports the branch. A detached HEAD still reads HEAD. * review: silkd docs: attach never ends in done, a dropped port_forward can linger * review: silkd and sandbox-init borrow what they only read claim_env keeps its one read buffer and parses into borrowed pairs, so an exec, pty or session drops 2N String allocations. BootCfg and IpParam borrow the cmdline, read_manifest returns its text, apply_config sets literals, pty and upload signatures take plain borrows, and scan_serials formats a path only when it reads it. Tests changed only where a literal's type or a temporary's lifetime named the old owned form. * review: silkd drops code that cannot run Child::id is Some right after spawn, a session body is never empty, Path::file_name is never empty, abort on a finished task is a no-op, and the exec rebinds did nothing. Table::new and Broker::new only aliased Default; git hand-rolled error_frame; the imds consts and handle, DEFAULT_INIT and PROTO_VERSION had no outside user; the ignored bench asserted nothing. * review: silkd statics and test helpers follow the file layout Function-local statics and consts move to the top-of-file block, and the one-caller proxy_vars and silkd_net wrappers go with them; test helpers move above the tests and private below public. * review: silkd comments shrink to what the code cannot carry Module narratives go to one line, the imds token fact moves to its const, a history comment goes, make_controlling_tty becomes safe because it has no memory precondition, and find's at_root stops shadowing the root param. * review: silkd tests use the shared request and assert_error helpers * sdk/python: ship the PEP 561 py.typed marker The package is fully annotated and mypy-strict, but the wheel carried no marker, so a consumer's type checker treated every cocoonsandbox import as untyped (reveal_type gave Any). Add the marker and the setuptools package-data entry; a scratch wheel now lists cocoonsandbox/py.typed and mypy on a consumer reveals cocoonsandbox.sandbox.Sandbox. * review: one-line docstrings where several lines restated the signature or sibling docs * review: drop duplicate test fixtures and a subset test; share FakeConn through conftest * review: tests before helpers in test_stream and test_client * review: adapters floor the SDK at 0.1.13, where the in-guest kill after a timeout cut first shipped * review: silkdtest's default arm calls serveStateless The arm re-spelled serveStateless byte for byte. * review: trim multi-line comments on unexported decls, capitalise own-line comments Own-line comments start uppercase per /code Comment Style; multi-line godocs on unexported names cut to one line. * review: guestserver keeps its report utility below the halfCloseSink block Layout only: /code File Organization puts utility funcs below the type's methods. * review: volumesmoke rejects non-local probes with filepath.IsLocal Same verdict as the hand-rolled path test on 15 inputs. * review: harness owns the guest proxy address, RelayConn and a package-level Request The proxy address was declared in four drivers. Dial returns RelayConn so portsmoke calls CloseWrite directly. PortRelay.Request never read its receiver. credsmoke's expectPlaceholder locals stop shadowing the entry and header consts. * docs: deploy's upgrade notes name the release each starts from At v0.1.21 "this release" read as every upgrade needing a drain, a fresh data dir and an emptied bucket. Releases come from the first tag holding each commit: #81 v0.1.6 (the pool key's engine axis went), #364 v0.1.18, #59 v0.1.5, #69 v0.1.5 and #72 v0.1.6, #171 and #169 v0.1.11. * docs: deploy and egress samples load and name a pulled image by its full ref A pulled image is keyed by its whole ref, so template base:24.04 fails the golden build with image not found, as the README already fixed in #425. The deploy sample's cluster_key and the egress sample's missing egress_ca failed config.Load. * docs: deploy states what the code requires of a failed claim, a golden and a dump A wake of a failed egress-lane or volume claim answers 409, tenants get 403 on two more root routes, golden dumps queue, an idle default acts on running claims at the next sweep, and an adopted replica always wakes on a read; none of these had a line. * docs: deploy gives the archive registry's DDL A role without CREATE could not enable archive_failover from the docs: the other three shared tables have a pre-create DDL and this one had none. * docs: deploy keeps the S3 layout and the replica bound in one place each The checkpoint_store row repeated the Reclaiming S3 storage section, which takes the facts only the row carried (lineage scope, marker order, re-stamped manifests); the TTL, replica-loss, tenant-outage and cluster-invariant passages become links to the pages that own them. * docs: cluster lists registered volumes and the TTL, and says when the digest covers secrets An operator rebuilding data_dir from config.json lost runtime-volumes.json silently, and gauge 0 read as a check of two secrets a keyless digest never covers. * docs: security names the TLS edge, the re-admitted prefix and the shared usage table Stated as hard constraints these ruled out what deploy ships: a TLS edge for SDK clients, egress_internal_allow, and meta_store.usage. * docs: benchmarks and performance keep each caveat once * docs: api states redirects, operator tokens, fork env and the checkpoint list as the server does The mint endpoint never answers the 503 and 429 its paragraph listed: the browser request does. A template another node deleted stays in tplSet, so the cluster page's never-fires claim was false for that entry. * docs: api lists the errors of a call that reaches a guest once Wake, agent, ports and exec carried the same four tails. Restore failures, 409s, release, hibernate and the preview listener differ, so they keep their own lines. * docs: sdk, silkd, langchain and android match what the code does Go Exec runs the argv given, with no implicit shell; every control-plane verb returns APIError; the checkpoint option refusals and a fork child's env were incomplete. * docs: the sdk pages link the volume and deadline rules the api page owns Go-specific and Python-specific names stay. The settled-entry sentence stays on the Go page, its only source. * sandboxd: a preview listener that cannot bind is fatal A taken preview_listen was logged and ignored: the node stayed healthy while the preview endpoint kept minting URLs for a dead port. The main listener already exits on the same failure, and systemd Restart= is the retry loop. http.ErrServerClosed stays clean. * tenants: the contract replaces the set with no tenants An empty Replace must delete every tenant on both backends: pgx sends a nil slice as NULL, so a Collect-built name list would delete nothing on pg. * review: one owner for the max fork default, the routable-host test, the s3 store test and the tenant claim mapping config.DefaultMaxForkCount replaces the second literal in pool, config.RoutableHost replaces the copy in server.New, Config.S3Checkpoints replaces three spellings of the kind test, and forTenant replaces the tenant-to-options mapping written in both claim paths. Behaviour-preserving, net zero lines. * review: store package docs fit the comment budget, constant errors use errors.New Package docs of store, s3 and peer shrink to one line (docs/deploy.md owns the key layout); the SweepStaging godoc and the redirectCacheEntry comment restate their names; three message-only fmt.Errorf become errors.New; VolumePatch is decode-only, so its slices follow the omitempty rule. * pool: a fenced node captures no replicated claim Fork, Checkpoint and Promote reached captureAllowed with no fence gate, so a node past its lease window could hand out children, a checkpoint or a template from a claim a peer may already own. ErrLeaseLost maps to 503 through the error table and is benign to the archive and replicate sweeps. * pool: consumeArchive reads the row epoch under the manager lock An unfence rewrites ArchiveEpoch holding only m.mu, and a release or replica retirement read it holding only Transition, so the two raced. The stale read cost at most a refused Consume and a settle round trip. * review: pool comments name what the code does now Two field comments named fields that moved to configView, checkTemplateOwner never deletes, HasPromotedTemplate's cache can go stale, and poolSeedHash's field list drifted against the lines below it. * review: checkpoint.go keeps const and var above its types * review: fold pool helpers with one caller archiveDeleteFor, unpooledTemplates, archiveFetchErr, dropStaleRow, hasRegularFile, deliverGuestEnv and TenantRecords each had one caller. reputArchive's held parameter was dead: its only caller guards a non-zero epoch. * review: one owner for pool facts stated twice replicaPools repeated view.poolReplicate, whose key set a reload cannot change and PUT /v1/pools cannot set. The view's map guards bought nothing, since every reader takes the zero value. env and replicable spelled out runningShaped, AgentSocket spelled out isFenced, and the two reader teardowns stripped a cancel that removeVM strips itself. addClaimLocked gains one atomic load on nodes with archives and nothing elsewhere. * review: pool sweeps and wakes carry what they have commitWake logged its persist failure at Warn and its caller rebuilt the error with the cause dropped; it now returns the error. idleOnce copied id and token so idleHibernate could look the claim up again, and hibernateLocked's setIfLive already answers ErrUnknownSandbox for a released claim. Fork and Checkpoint no longer wait on Transition for a usage field, since every write of VMName after publication holds m.mu; lockedVMName stays for tests, which read under Transition. * review: types.go splits into env.go and volume.go Pure move: the env types, limits and patterns go to env.go and the volume types, modes and mount validation to volume.go, matching the existing env_test.go and volume_test.go. envView and namedVar stay beside the Sandbox that owns them. No line changes beyond imports and gofmt alignment. * egress: an upstream dying mid-body truncates the guest's response relay dropped the copy error, so net/http wrote the terminating chunk and the guest read a partial body as complete. Hijack and close the conn on a copy error, which leaves the chunked stream unterminated. Success costs one nil check. * outbound: ArmProxy closes a displaced door before it binds The close unlinks the socket path, so a re-arm that bound first left the guest with no door while Route reported relay. A re-arm with no policy left now closes the door too, which makes the Disarm in restartLocked dead. The claim path pays one map delete under a lock it already holds. * engine: a restore gets the dump budget A dump may take 15 minutes while an eager-copy restore of the same image had 2, so a slow page cache could fail the wake it had just made possible. cocoon gives vm.restore the same limit as vm.snapshot. Clone and cold boot keep the command budget. * review: drop checks and fields the code already guarantees ReadSlice caps a line at the reader's buffer, IsGlobalUnicast already excludes link-local and multicast, UnixListener.Close already unlinks, marshalCA's public key is its PEM key's, and interDER copied interCert.Raw. * review: DialFunc's Router methods serve only tests The only production egress.New takes a claim as its router, so the two methods move to the egress test file. * review: a CA key error names the key its caller parsed parseECKey said intermediate key while parsing the root, so IssueIntermediate reported root key: intermediate key. validHostName takes the package's maxHostLen for the same DNS limit. * review: cut a restating godoc and fold the engine package doc to one line * review: layout: silkd helpers live in silkd.go, tcpAddrPort beside proxy.go's other address helpers installca.go kept the generic guest write and exec helpers every other engine file uses; installOutputMax becomes guestOutputMax since it caps every guest exec. * docs: name the capture fence, the restore budget, a preview bind failure and a cut upstream * review: VolumePatch keeps the SDK's omitzero tags, as the wire parity test requires * engine: a clone restores memory like a restore, so it gets the dump budget cocoon's vm clone and vm restore share restoreVM and its 10 minute memory transfer, and an archived wake and a fork go through clone. A hung refill clone now holds its slot for up to 18 minutes, which stays bounded and loud. * review: silkd's language server feed needs no flag or channel, and one helper sets the claim env * review: refill.go keeps the refill loop; golden builds go to golden.go, VM provisioning to provision.go, destroyAll and dropSnap beside destroy, runBounded to pool.go with its callers across the package * review: the reaper leaves claim.go for reap.go, since expiry handling is not claim admission * review: claim and authed resolve a credential, so they sit in operator.go with resolve and byID * review: the pool type joins its watermark methods in watermark.go, so its type block is atomic * review: rewriteArchiveRow writes an archive row, so it sits in failover.go beside refusedReplace * review: NewManager delegates template and volume seeding to loadTemplates in template.go and seedVolumes in volume_catalog.go, keeping the order * review: the catalog owns its entries and holders, so catalogVolume, catalogSnapshot and the holder bookkeeping move to volume_catalog.go as volumeCatalog methods * review: the seven single-flight sweep guards group into one sweeps field on Manager * review: recordLocks owns the per-record lock maps and their mutex, and recLock's doc says it returns the lock without taking it * review: healVetoes owns the heal veto maps and their mutex, replacing the three Manager methods and fields * review: ckMarkers owns the archive delete markers, and the delete and retry chain moves with it to archive_delete.go * pool: an uncordon during an evacuation's drain stops it Evacuate waits for the warm VMs to go before it starts its run. A DELETE /v1/drain in that wait found nothing to stop, and the run then moved every claim off an uncordoned node. * pool: a tier claim woken past its lease before the export gets a fresh lease The reaper archives a hibernated claim of a tier pool at its lease end, and the same call a moment after the export restores it with a fresh lease. A call in between woke it with the past deadline, so the next reap tick destroyed it. * pool: a running claim that moved to a peer closes its egress doors forgetAdopted removes the VM with no sandbox id, so the removal never disarms. The doors, their listeners and their goroutines stayed until the next restart, one set per moved claim, unlogged. * docs: a moved claim's token, its first wake, and the 503s an evacuation and the tenant store answer * sdk: Close follows a sandbox that moved off a retired node Close looked the sandbox up only on a 404 from its old owner. Once that node is retired, the first release fails to connect, so the claim stayed on the adopter until its deadline, and every retry hit the same dead address. A connection failure now looks it up too, and keeps its error when no node answers.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Started as three lifecycle gaps found by a close-repo review; a whole-design
audit of all 392 tracked files then turned up several more, including two
tenant-isolation holes that were runtime-proven, not inferred. Everything below
is on this branch.
Isolation and cluster correctness
PoolKey.Hash()has no tenant axis andresolveGoldendecoded the templaterecord only to discard
rec.Tenant, so promote and delete were tenant-scopedwhile the read path was not. Proven: acme promoted
acme-private, betaclaimed the same key and got back acme's content digest — beta booted acme's
snapshot. A foreign template now resolves as absent, byte-identical to a name
that was never promoted, so existence stays unconfirmed. Root-promoted
templates stay shared, as pools are.
egressblock inherited its pool's whole allow-list,secret injections included, so adding a tenant token handed out the
operator's credentials. Both single-sided branches now deny:
pool ∩ tenant,with a missing policy on either side denying all; root has no tenant layer and
takes the pool policy whole. Verified first that
ok=falsereally denies —no listener means the egress-lane NIC stays nft-locked and the none-lane dial
is refused. Upgrading: a tenant that relied on inheriting must now declare
its own block.
forgetdropped it, butmergere-inserted any node it did not already know, so one push/pull from a peer
that had not yet seen the death resurrected it — permanently, since SWIM never
re-fires
NotifyLeave— and re-gossiped it cluster-wide. SWIM membership isnow the authority:
mergeaccepts gossip only about nodes it currentlyreports. (An earlier epoch-tombstone commit had two residuals of its own —
memory-only, and entries that never freed — which this replaces.)
Lifecycle
/devnode, not just the sysfs serial: the kernelpublishes sysfs first, and
resolve_disksmounts with no retry. Same class asthe settle race measured at 3.5% of four-volume claims on bare metal — this
was its third instance, alive in
boot/initfor six weeks.fs_watchoverflow signals by dropping the sender rather than setting a flaga parked loop could miss; the buffered ordered prefix is delivered before the
terminal error.
PreviewDial, andwith it
wakeResolvedand theTransitionlock that serializes a claimagainst its own hibernation. Every request dials again.
idle_hibernate_seconds(the lane refuses to hibernate,so the setting could only ever fail), and the sweep skips egress claims of
unpooled keys, which the config gate cannot reach.
Pty.Closeis idempotent and unblocks an unread reader.Supply chain and gates
rust-toolchainsaidstable, so thekernel build,
sandbox-initandsilkdwere reproducible only by luck. Basesare pinned by digest and both toolchains to the version the Linux gate runs;
dependabot owns refreshing them, because a pinned digest goes stale
silently.
make sh-lintplus a workflow.SDK surface
APIErroris exported with its status: the SDK's own redirect walk brancheson it, so callers need it too.
WithHTTPClientfor a caller's own transportor deadline; no blanket client timeout, because checkpoint and promote block
for as long as the snapshot takes.
engineis a pool-key axis the SDK never carried, so anInfo→SetPoolsround trip — a declarative full replace — silently drained every
fcpool.The two pool gauges omitted it too, so two pools differing only by engine
emitted one label set and the scrape failed.
claim_refand the listing that reads it back now exist on both SDKs; Pythongains
attach,drain,uncordon(the OpenAI adapter was hand-building aSandboxaround the missingattach),ExitError.stdout,Pty.exit_code,and
Watcher.error. Go gainsCheckpoint(id)— MCP was listing everycheckpoint on the node to resolve one id.
session outliving it lost every sandbox mid-conversation: it now claims for an
hour, accepts
net/size, and releases what it claimed when the session ends.Docs
Eleven factual corrections against the code (the
engineaxis, the/metricsinventory,
secretsandegress_internal_allow, the keyed HEAD probe's 401,the audit op name, which verbs are connection-bound, volume discovery scope),
the node-operation verbs
cluster.mdtold operators to use butsdk.mdneverdocumented, and the checkpoint-delete lifecycle reduced from three copies to one.
Validation
make go-lint(golangci run + fmt, dual GOOS, 5 modules) — 0 issuesasl ./...dual GOOS across the Go modules — 0 findingsgo test -race -count=1 ./...— green in every modulecargo fmt --check,clippy --all-targets -D warnings,cargo testforboth Rust crates, in a Linux container on the pinned 1.97.1 toolchain
(silkd 13 suites, boot/init 17 tests)
ruff check+pytest(153) for the three Python packagesmake sh-lint— 0 findingswritten as a throwaway test, run against this branch, and deleted
source (v0.5.4
net.go:1302-1307): membership merges, firingNotifyJoinsynchronously, before the user-state delegate
Hardware round on the final branch (.79 bare metal, main@8f615d5 vs
branch@64c1aff, logs in
~/pr81-ab/logs/):daemon lives, 16 samples each, every sample warm-tier-proven, 0 flakes — main
p50 0.489/0.488 ms vs branch 0.482/0.475 ms; |A−B| is inside the A/A drift.
The tenant-gated template resolution and the egress fail-closed change cost
nothing measurable.
gate costs +4 µs in the
resolvephase (p50 83→87 µs); total init→handoffp50 1387→1352 µs, branch faster within noise.
per-request re-dial 0.427/0.447 ms = +0.21 ms/request — the deliberate,
documented cost of per-request revocation/wake serialization (see the
preview commit).
One transient e2e suite failure was observed once during development and
analyzed: fail-fast shape (suite finished 1 s faster than a pass), occurred
while a second
go test -raceof the same package ran concurrently in the samecheckout, and did not reproduce in 87 subsequent runs across idle, CPU-loaded,
and faithfully-reproduced concurrent conditions, with CI green on the same
commit. Treated as environmental; the two tests that own the suite's wall time
(both waiting on the refill ticker) are where to look if it ever recurs.
Test-hygiene findings from the audit (13 of 79 Go test files, silkd's untested
git_push/git_pull, the Rust response-encode fixtures) are deliberatelydeferred to a follow-up PR.