You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 79ab31a
Browse filesBrowse the repository at this point in the historyBrowse files
fix(native): keep foreground recovery responsive after publisher exits (#739)
Foreground publisher loss now keeps services observable and stoppable
while exit proof waits for a busy provider lease. Retaining recovery
also recognizes a reused macOS frontend PID by birth time, so the
replacement process does not strand the old run's finalization.
This follows #719 and addresses its remaining review findings:
- Start publisher notices only after readiness; an early notice cannot
hide the real startup diagnostic or interrupt cleanup.
- Dispatch owner wake decisions through one tested step. Publisher exit
keeps services up and port status down; later signal or authenticated
down remains available.
- Separate observed exit from committed proof. Reap without blocking the
owner loop, retry pending proof, and commit it before guest cleanup.
During contention, ps can report authenticated host ports as down
without guest service or readiness claims.
- Keep restart following for healthy sibling services, while excluding
exited publishers from both bridge replacement and probe rearming.
Replacement readiness checks the selected child; first startup still
requires all publishers live. Port-conflict handling requires that
replacement's committed exit and occupied port. Losing one publication
conservatively pauses following for all publications of that service.
- Record optional macOS frontend birth times; share the exit predicate
across down, recovery, waiting and lifetime inspection. Legacy records
remain PID-only. Re-read acknowledgements after observing exit and
throttle birth inspection.
- Identify detached static refusals with a checked frontend-only stage:
say nothing was started, name the configuration fix and preflight
command, and retain recovery guidance for later failures.
- Preserve provider-pin reasons through ps JSON, detached up and exec.
Give unreadable publisher identities their own waiting/retry reason.
Teardown outside the signalled lifetime fixture stops its exact
publisher even after fatal SIGHUP.

Validation: Bun 1.4.4 typecheck, lint/privacy, staged leak scan, CLI
reference and current CLI build; focused source-CLI regressions; Rust
1.97.1 default checks/tests, feature Clippy, relay contract, real
portable publisher processes, installed-candidate contract; macOS ARM64
feature cross-Clippy; registered TLA models. The publisher model
explores 168 states with seven positive/negative/reachability
configurations. The owner decision mutation fails on the old cleanup
behavior.
Evidence limits: Linux and cross compilation do not run the macOS-only
reaping/lifetime tests or qualify a VM. The portable owner-loop
regression covers production decision dispatch, and native child tests
cover reaping; replacing the dispatched effect body still needs native
integration coverage. The terminal fixture retains a narrow setup window
before its identity marker. Cross-provider recovery remains out of
scope; latest provider pin is retained.
Real-VM check: kill one publisher while the provider lease is busy,
confirm ps marks its port down, restart healthy loopback and routed
siblings without graph cleanup or dead-publisher resurrection, then stop
via Ctrl-C/down; also repeat the reused-frontend-PID recovery and
group/pty/terminal-loss lifecycle cases.
The full local gate before the base integrations passed all 117
TypeScript files, default Rust library tests (934 passed, 37 ignored),
feature library tests (955 passed, 46 ignored), relay tests (34),
installed-candidate contracts (4), feature Clippy, macOS ARM64 feature
cross-Clippy, typecheck, lint, staged leak scan and registered models.
The portable replacement-readiness negative control fails under the old
global-exit rule. The new macOS two-child test exercises real reaping
with a substituted publication read; it does not exercise Engine or
guest state. The one-service restart model does not cover cross-service
readiness.
Merged `origin/next` through 26a1429 in normal merge commit
830387d. Both model-coverage rows are
retained; #729 setup and #732 log follow integrate alongside the
lifetime changes. #737's validated detached handoff carries #739's
provider-pin reason; #737 static preflight and #738 detached readiness
remain intact. Static-start provenance comes only from the frontend
review, survives safe wrappers, and is checked across IPC; runtime error
output cannot mark a failure as static. Regressions cover the same
refusal code before and after start, malformed phase metadata and a
cold-pool source-CLI UDP refusal with no boot request. The same
persistent reviewer is clean on the exact committed tree. Final focused
integration run: 361 passed, 16 skipped across 11 files; typecheck,
lint, privacy and staged leak scan passed. Hosted CI and Codex at this
new head remain pending; #733 may select a test subset. Lead is running
the equivalent local merge on Mac (ctrl-c, kill-publisher, hup,
up-twice, up-partial, webdb, recover-old).
<!-- codesmith:footer -->
---
<a
href="https://backend.blacksmith.sh/track/run-proof?expires=1797098296&installation_model_id=17053&pr_number=739&ref=codesmith_pr_footer&repository=hack-dance%2Fhack&return_to=https%3A%2F%2Fgithub.com%2Fhack-dance%2Fhack%2Fpull%2F739&variant=proof_this_pr&signature=56e5892802de151e0d054597bf28d494ae100896f4a525cc4afca4d038d47c51"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://pr-comments-assets.blacksmith.sh/codesmith/proof-proof-this-pr-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://pr-comments-assets.blacksmith.sh/codesmith/proof-proof-this-pr-light.svg"><img
alt="Proof this PR"
src="https://pr-comments-assets.blacksmith.sh/codesmith/proof-proof-this-pr-dark-3x.png"
width="121" height="28"></picture></a>
<sup><a
href="https://app.blacksmith.sh/hack-dance/codesmith/hack/pr/739?autoLogin=true&ref=codesmith_pr_footer">Open
in [code]smith</a> · Need help on this PR? Tag
<code>@codesmith-bot</code> with what you need.</sup>
<!-- codesmith:autofix:disabled -->
<!-- /codesmith:footer -->
---------
Co-authored-by: Kit <344842+roodboi@users.noreply.github.com>
Copy file name to clipboardExpand all lines: docs/reference/v5-model-coverage.md
+3-3Lines changed: 3 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -37,7 +37,7 @@ must be attached to the eventual exact integration revision.
37
37
| Property and critical states | Model coverage | Implementation/regression boundary | Remaining qualification |
38
38
| --- | --- | --- | --- |
39
39
| Native log follower versus lifecycle mutation, frontend loss, stopped container and transport loss | No new state model: the follower observes without a mutation lease and grants no cleanup authority | `provider/graph/log_follow.rs` and its native counterpart pin Ready admission, container and start generation; `provider/engine/log_follow.rs` bounds framing and distinguishes real EOF from stopped-service idle cutoff. `lifecycle_log_follow_tests.rs` keeps the production transport on a fake Engine socket active while acquiring startup and operation leases, with a same-lock contention control and output while each lease is held. `native-log-follow-source.test.ts` guards the observer-only route; process tests cover cancellation, frontend loss, truncated/partial output and idle end reasons | 4 KiB envelopes, 8 KiB wire lines, 64 Ki-character retained line prefixes. The portable lease test fakes identity/state checks; it does not exercise host process/disk auditing, graph admission or the VM relay. Real relay EOF on stop/down/restart, VM loss and concurrent lifecycle latency require Apple Silicon acceptance. No performance or whole-product parity claim. |
40
-
| Foreground group interrupt; publisher exit before authenticated stop; dead owner's pending cleanup enrollment and bridge release | native-publisher-stop | [Process adapter](../../src/backends/native-project-process.ts) isolates Compose owners regardless of hooks; Rust `process::publisher_command` isolates publishers; `legacy_cleanup_enrollment` binds the old coordinator and inventories before a distinct recovery intent. [Group regression](../../tests/native-foreground-group-interrupt.test.ts) signals the whole group for fresh/restore owners; Rust real-publisher process, frontend-loss, exact child-reaping and enrollment tests check grouping, lifetime wakeups and stale selectors. | One graph, one publisher, one crash, one identity replacement and one audited successor boot; 110 states. SIGHUP requests retaining cleanup; a reaped publisher enters an observable publisher-down state until explicit stop. Native process tests and frontend framing/ps tests cover signal delivery and notices. Dependency release is a separate post-completion step; `NoStrandedReservation` rejects success with a held reservation. Real reservation-file regressions check retry, capacity reuse and refusal fences. Separate stopped-receipt and intent-completion commits are collapsed; TypeScript dispatch and Rust completion-digest regressions cover that interruption window. Publication-registry sync/rename is abstracted and covered by portable commit-interruption tests. Kernel identity, socket/inode proofs, guest observations, durable storage and existing next.13 home recovery remain implementation/Apple Silicon VM gates. No eventual recovery is asserted. |
40
+
| Foreground group interrupt; publisher exit before authenticated stop; dead owner's pending cleanup enrollment and bridge release | native-publisher-stop | [Process adapter](../../src/backends/native-project-process.ts) isolates Compose owners regardless of hooks; Rust `process::publisher_command` isolates publishers; `legacy_cleanup_enrollment` binds the old coordinator and inventories before a distinct recovery intent. [Group regression](../../tests/native-foreground-group-interrupt.test.ts) signals the whole group for fresh/restore owners; Rust real-publisher process, frontend-loss, exact child-reaping and enrollment tests check grouping, lifetime wakeups and stale selectors. | One graph, one publisher, one crash, one identity replacement and one audited successor boot; 168 states. Publisher exit observation is separate from lease-bound reaping/proof commitment; `NoUncommittedExit` rejects owner cleanup while proof is pending. A busy lease stutters without delaying stop control. SIGHUP requests retaining cleanup; a reaped publisher enters an observable publisher-down state until explicit stop. Native process tests and frontend framing/ps tests cover signal delivery and notices. Dependency release is a separate post-completion step; `NoStrandedReservation` rejects success with a held reservation. Real reservation-file regressions check retry, capacity reuse and refusal fences. Separate stopped-receipt and intent-completion commits are collapsed; TypeScript dispatch and Rust completion-digest regressions cover that interruption window. Publication-registry sync/rename is abstracted and covered by portable commit-interruption tests. Kernel identity, socket/inode proofs, guest observations, durable storage and existing next.13 home recovery remain implementation/Apple Silicon VM gates. No eventual recovery is asserted. |
41
41
| Exclusive claims; competing writers; partial owner publication | graph-admission, registry-writer, publication-journal, missing-publication-lock, native-frontend-recovery | Admission/registry locks, pending publication and exact saved owner; related runtime ownership tests | Real filesystems, boot/incarnation identity and unsupported writers remain independent. |
42
42
| Interrupted native restart: Ctrl-C or a frontend crash while the old run stops; `up` and `restart` refusing each other | native-restart-intent | The restart's stop requests run in their own process group (`processGroup: "own"` in [native-runtime-client](../../src/backends/native-runtime-client.ts)); a resumed `prepared` intent accepts a graph its owner stopped ([restart](../../src/backends/native-project-restart.ts), [preflight](../../src/backends/native-project-restart-preflight.ts)); [restart regressions](../../tests/native-project-restart.test.ts) send a real group SIGINT and cover the resume; HACK-2035 regressions force SIGINT/SIGTERM child rejection before frontend abort in preflight and down, using the runtime client’s preserved `stopSignal` | The finite model abstracts interrupt delivery as one action and does not prove child-exit versus frontend-handler ordering; deterministic tests cover that classification. A stale restart lock after a frontend crash; down hooks still share the terminal's group; physical guest acceptance. |
43
43
| Foreground Compose startup failing after resources started; unconfirmed cleanup; unauthenticated final inspection | native-startup-owner |`recordFailedStartupOwner` in [native-project-start](../../src/backends/native-project-start.ts) publishes the authenticated failed run with a finalization lifetime; [start regressions](../../tests/native-project-start.test.ts) cover recording, an unauthenticated receipt and a retained mapping | A refused final inspection still leaves only the runtime journal; crash between finalization registration and mapping publication; physical guest acceptance. |
@@ -49,7 +49,7 @@ must be attached to the eventual exact integration revision.
49
49
| Graph cleanup container stop batch: bounded in-flight stops, shared worker queue, connect-stage retry, batch-wide unreachable state, deadline expiry, worker panic | **Coverage gap:** no maintained model covers scheduling inside the stop batch; native-route-stop covers a different stop protocol | `dispatch` and `attempt` in [the stop batch](../../packages/runtime-core/src/provider/engine/stop.rs): at most 4 stops in flight, every worker joined; only `connect`/`connect_timeout` failures retry (25 ms doubling to 500 ms, within the batch deadline and 30 s of a stop's first connect failure); once one stop exhausts its window, later connect failures return at once; a deadline reached during backoff keeps the connect stage; `response`, `timeout` and `transport` never retry. Regressions in the same file's `provider::engine::stop` tests (bounded in-flight socket server, late socket, missing socket, 14-stop retry counts, persistent connect failures, oversleeping backoff, non-retried stages, worker panic) and `fourteen_service_connect_failures_keep_resumable_progress_and_retry_only_unstopped` | Tests cover exact attempt counts and the in-flight bound for fixed interleavings; no model explores all worker schedules or proves at most one delivered request per stop. Assumes a failed connect sent no request bytes and that the stop budget covers fully serialized grace periods. Graph-level resume after a partial batch is covered by tests only. Host qualification of the descriptor-pressure root cause (HACK-1302) remains open. |
| Shared host helper instance: overlapping ups, session adoption, stale create decisions, replacement before finalize, epoch-record failure, interrupted cleanup, concurrent down, displaced launches, failed down session stop | shared-helper-instance | `src/lib/project-shared-helpers.ts` (shutdown generation, `assertCurrent`, `SharedHelperSessionOwnership`); `src/lib/project-lifecycle-controller.ts` session decision, the locked creation transaction (revalidate, publish, create, environment, record, stop on failure, all under one lock), launches fenced by the lock and a token check, abort and signal cleanup through the same epoch decision (`runIfOwned`), an unknown exact-presence answer failing the inspection, and the locked conditional finalize; `stopSessionBeforeReleasingHelpers` (down, restart, native retirement keep holders when the session stop fails); `runMuxCommand` bounds every lifecycle and backend mux call so it settles before the lock is released, and reports a timeout as unknown (never absence); `tests/mux-command-timeout.test.ts`; `tests/project-lifecycle-controller-ownership.test.ts`, `tests/project-shared-helpers.test.ts`, `tests/native-project-down.test.ts`, e2e `shared-host-helpers` | Helper holder/rollback and claim protocol are not modeled (claim fixtures cover them); timeout settlement is an assumption the code does not guarantee (a killed tmux client's queued command may still apply, so a late kill or window command can reach a newer session; consequences and the session-ID fence, not in this slice, are in the README); zellij is refused for shared-helper lifecycle sessions (token written after creation); the gap between a down's session stop and its release (an up that captured after the stopping mark can still succeed there) is a documented first-slice limitation; real mux/process/lock-file behavior. |
52
-
| Engine restart policies: stop versus pending restart, guest engine restart, stale incarnations, terminal proof | service-restart-policy | Receipt-admitted policy, disarm-before-stop and `restarting` stop preflight in `shutdown.rs`; admission refusals for incarnation-bound delivery classes in `restart_policy.rs`; owner-driven `Rebind` after full retirement, fresh ownership/attachment checks and a new reservation in `bridges/restart_rebind.rs`; route probe sample-generation rearm; Rust tests for admission, endpoint identity, port-busy refusal, disarm and stop preflight | Real guest engine behavior (crash restart, stop during backoff, disarm before deletion) needs native host evidence; several services, backoff timing and retry counting are not modeled. Same-port/route reachability after a policy restart, route probe rearm and per-slot child retirement still require real-VM host evidence. Rebind abstracts retirement/reservation/publication; relay lifecycle, fence and staging models cover their separate failure boundaries. |
52
+
| Engine restart policies: stop versus pending restart, guest engine restart, stale incarnations, terminal proof | service-restart-policy | Receipt-admitted policy, disarm-before-stop and `restarting` stop preflight in `shutdown.rs`; admission refusals for incarnation-bound delivery classes in `restart_policy.rs`; owner-driven `Rebind` after full retirement, fresh ownership/attachment checks and a new reservation in `bridges/restart_rebind.rs`; route probe sample-generation rearm; Rust tests for admission, endpoint identity, port-busy refusal, disarm and stop preflight; portable selected-child readiness guard plus macOS two-real-child polling regression prevent historical sibling exits from rejecting a replacement or misclassifying its own listener | Real guest engine behavior (crash restart, stop during backoff, disarm before deletion) needs native host evidence; several services, selected-child readiness during sibling failure, backoff timing and retry counting are not modeled. Same-port/route reachability after a policy restart, route probe rearm and per-slot child retirement still require real-VM host evidence. Rebind abstracts retirement/reservation/publication; relay lifecycle, fence and staging models cover their separate failure boundaries. |
53
53
| Foreground Ctrl-C down hooks: concurrent `down`, second interrupt, crash between or during hooks, stop during `up.after`, next generation | native-foreground-down-hooks | Value-free compare-and-swap journal in [down hooks](../../src/backends/native-project-down-hooks.ts), foreground orchestration in [foreground stop](../../src/backends/native-project-foreground-stop.ts) and `hack down` in [native down](../../src/backends/native-project-down.ts); [journal](../../tests/native-project-down-hooks.test.ts), [foreground](../../tests/native-project-start-stop-hooks.test.ts) and [down](../../tests/native-project-down-journal.test.ts) regressions | Durable writes are atomic in the model; real fsync/power loss, a stale journal lock after a crash inside the lock, and the live process-group behavior of the graph owner on a host are not covered. One generation and at most two `down` invocations are explored. |
54
54
| Dependency rebind, automatic refresh, incomplete initializer, terminal reuse | dependency-rebind, dependency-auto-refresh, stopped-pool-startup, restore-history | Dependency claim/readiness, saved execution, restore history and source convergence tests | Actual installer failure, cache corruption, retained application state and resource plateau. |
55
55
| Private environment allocation lifetime: staged values and private file members, crash between record and stage, retirement with the container |**Coverage gap:** no maintained model covers environment or credential delivery; file members share the existing allocation, intent record and retirement, so the state machine is unchanged |`packages/runtime-core/src/provider/environment.rs` (preparation bounds, staging layout, verification), `managed_environment.rs` (envelope v2 refusals, forwarding), `graph/launcher.rs` (read-only binds, overlap refusals) and `environment_recovery.rs` retirement tests | Live VM evidence that only the owning UID reads a member, writes fail read-only and the member is absent after cleanup; refresh or rotation of delivered files is not implemented. |
@@ -73,7 +73,7 @@ liveness/reachability controls have their own documented assumptions.
73
73
74
74
| Model | Positive distinct states | Configurations |
0 commit comments