Skip to content

fix(runtime): confirm acknowledged publisher retirement during restart - #121

Closed
roodboi wants to merge 36 commits into
nextfrom
codex/confirm-acknowledged-restart
Closed

roodboi wants to merge 36 commits into
nextfrom
codex/confirm-acknowledged-restart

Conversation

@roodboi

@roodboi roodboi commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

A retained restart with explicit frontend recovery could fail after ordinary acknowledged shutdown: normal foreground shutdown had already removed the publisher endpoints, but the recovery command selected an older crash-cleanup receipt.

The recovery command now confirms current-boot acknowledged cleanup under the publication lock and a fresh Engine lease. It validates the exact stopped receipt, cleanup effect and durable acknowledgement, absent compute, retained volumes and absent publisher endpoints without changing graph or publisher files. Exact current recovery completions retain their existing witness-aware path; prior-boot proof remains in the existing recovery path. Invalid current enrollment refuses without borrowing authority from historical sidecars. Acknowledged retirement and dependency release now use the same publication-before-Engine lock order as foreground restore.

Validation: default Rust suite 1,012 passed / 62 ignored; all-feature suite 1,101 passed / 84 ignored. Rustfmt, strict Clippy for both feature sets, staged privacy scan and diff checks passed. Regression coverage includes current versus historical completion, old boot and incomplete current enrollment; the existing ignored native two-graph fixture now checks idempotent public confirmation, wrong-owner refusal, retained data and unchanged sibling state. That extended fixture has not been executed in this qualification. An initial full default run encountered a socket fixture WouldBlock; its log was preserved and the subsequent complete run passed without changing that assertion.

The signed M3 bundle passed live confirmation twice and a wrong-owner negative control: four retained volumes, exact graph/history/sibling/CA bytes and the retained VM owner/disks remained unchanged. Admission passed. The normal retained Event Agent restart now reaches ready-observed with nine health-checked services healthy, and verified HTTPS returns 200 for health, sign-in and all 17 rendered static chunks over HTTP/2. Chrome renders sign-in without a certificate warning. The restart used valid short-lived QA role credentials in process memory for the host hooks because Pulumi profile SSO refresh failed; no credential files or application changes were made. The foreground CLI remains the graph owner. Normal port-443 OAuth and signed-in browser acceptance remain open. All eight hosted CI checks passed for exact head 4a9ac29. Targets protected next and includes the recovery/admission/preflight stack through #120. This is a correction; no application repository changes or publication.

hack-cli-tests added 30 commits September 29, 2026 21:41
Stand-in invocations ran concurrently (a foreground `up` while the driver
polls `ps`) and each did an unlocked read-modify-write of state.json. A
`ps` that loaded the state before the second `up` saved its foreground
token then saved its stale copy over it, so `up` saw its token gone and
exited 0 before readiness; lost call records failed the acceptance test
as well. That failed 26 of 40 local runs, and Runtime state models on CI.

Each invocation now holds an exclusive flock across its read-modify-write
and releases it only before its long waits (the foreground loop and the
ps-hang fault). 40 of 40 local runs pass.
A review of #109 found two startup ownership gaps.

The native HTTPS owner discarded the identity listenPublishedUnixSocket
returned and only recorded one after lstat, chmod and lstat of the path.
A failure after publication then skipped listener and endpoint cleanup,
and a replacement in that window could be chmodded or recorded as the
endpoint. The owner now keeps the published identity at once, compares
its first observation against it, and never changes the endpoint by path.
Failure cleanup always closes the listener, which can no longer unlink
anything but the retired staging name, and then removes the endpoint only
while it is this listener's inode; a replacement is kept.

The helper awaited listen() outside its cleanup, so a listen that bound
the staging name and then failed could leak that socket and listener.
Listening is now inside the cleanup, and a failed start removes only a
staging socket this user created. The helper also sets the endpoint mode
(0600) on the staging inode before publication, so the MCP backend, the
HTTPS owner and the owner challenge no longer chmod the public path.

Controls: bind-then-fail, mode-preparation failure, an endpoint at the
AF_UNIX path limit (and one byte over: refused cleanly on Bun 1.3.9,
bound in full on 1.4.2, never truncated), and an owner fault or
replacement between publication and first observation, on Bun 1.3.9 and
1.4.2.
The private-bind fixture hooked chmod on the public mcp.sock to prove the
socket is private before its mode is set and that the creation mask is
restored. Since the endpoint's mode is now set on its staging socket
before publication, the hook never fired and the check failed. It now
observes that staging chmod with the same privacy and mask checks,
requires mode 0600, and fails if the published endpoint is ever chmodded
by path.
…a socket

A second review of the Unix-socket publication found three staging-path
windows where an entry this attempt could not prove was its own could be
changed or removed:

- After an ambiguous partial bind (no recorded identity), cleanup
  unlinked any same-uid socket at the staging name.
- The mode was set by path after an awaited lstat, so a replacement in
  that gap could be chmodded before the postcheck refused it.
- On success the staging name was unlinked unconditionally after the
  awaited link and endpoint lstat.

The socket is now created with exactly its mode (the umask during the
synchronous bind), so nothing is ever chmodded, and its identity is
recorded in the same tick. Publication links only while the staging name
is still that socket, and retirement removes it only while it is (check
and removal back to back). An entry that is not this socket, or cannot be
proven to be, is never removed: on failure it is moved aside while the
server closes, so the runtime's close-time unlink by name cannot reach
it, and then put back with the same inode.

The owner challenge's chmod dependency becomes an afterOwnerSocketPublish
test seam, and the private-bind fixture now requires the socket to be
created private with mode 0600 and no socket name to be chmodded.
…ng entry

The failure close moved an unproven staging entry to one random holding
name: if that name was occupied the close went ahead unprotected, and a
holding entry created between the check and the rename was overwritten.
The entry is now moved with link(2), which never replaces a holding
entry, over a bounded list of fresh holding names, and the staging name
is dropped only while it is still the linked entry. An entry that cannot
be moved aside (every holding name occupied, or not hard-linkable) leaves
the close to proceed, now documented as a residual.

The helper's documentation now states its scope: accidental and
concurrent entries in a private directory, not an adversarial process of
the same user, with the remaining path-based and post-return residuals.
hack-cli-tests added 6 commits September 30, 2026 15:26
Add explicit selector-bound recovery for a dead legacy HTTPS owner beside an unpublished shared-owner configuration. Preserve original inodes and the CA with resumable exclusive archival, held IPv4/IPv6 guards, and refusal on live or uncertain evidence.
Keep strict receipt identities as the default. Allow only an explicitly selected source-device witness and both current HTTPS inodes, with unchanged inode numbers and matching old/current device mapping. Recheck current owner, host and guest boot, source share and stopped graph scope before archival; preserve historical bytes and report original host-volume continuity as unproven.
@roodboi

roodboi commented Oct 4, 2026

Copy link
Copy Markdown
Contributor Author

Superseded by merged #122. This PR’s source was incorporated through #122, squash commit cf4b6e9. Independent acceptance remains in Linear. Closing as superseded; branches and worktrees are retained.

@roodboi roodboi closed this Oct 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant