Skip to content

Commit 79ab31a

Browse files
authored
fix(native): keep foreground recovery responsive after publisher exits (#739)
Foreground publisher loss now keeps services observable and stoppable while exit proof waits for a busy provider lease. Retaining recovery also recognizes a reused macOS frontend PID by birth time, so the replacement process does not strand the old run's finalization. This follows #719 and addresses its remaining review findings: - Start publisher notices only after readiness; an early notice cannot hide the real startup diagnostic or interrupt cleanup. - Dispatch owner wake decisions through one tested step. Publisher exit keeps services up and port status down; later signal or authenticated down remains available. - Separate observed exit from committed proof. Reap without blocking the owner loop, retry pending proof, and commit it before guest cleanup. During contention, ps can report authenticated host ports as down without guest service or readiness claims. - Keep restart following for healthy sibling services, while excluding exited publishers from both bridge replacement and probe rearming. Replacement readiness checks the selected child; first startup still requires all publishers live. Port-conflict handling requires that replacement's committed exit and occupied port. Losing one publication conservatively pauses following for all publications of that service. - Record optional macOS frontend birth times; share the exit predicate across down, recovery, waiting and lifetime inspection. Legacy records remain PID-only. Re-read acknowledgements after observing exit and throttle birth inspection. - Identify detached static refusals with a checked frontend-only stage: say nothing was started, name the configuration fix and preflight command, and retain recovery guidance for later failures. - Preserve provider-pin reasons through ps JSON, detached up and exec. Give unreadable publisher identities their own waiting/retry reason. Teardown outside the signalled lifetime fixture stops its exact publisher even after fatal SIGHUP. ![Foreground publisher shutdown statechart](https://raw.githubusercontent.com/hack-dance/hack/fix/native-publisher-lifetime-followup/docs/reference/native-publisher-stop.svg) Validation: Bun 1.4.4 typecheck, lint/privacy, staged leak scan, CLI reference and current CLI build; focused source-CLI regressions; Rust 1.97.1 default checks/tests, feature Clippy, relay contract, real portable publisher processes, installed-candidate contract; macOS ARM64 feature cross-Clippy; registered TLA models. The publisher model explores 168 states with seven positive/negative/reachability configurations. The owner decision mutation fails on the old cleanup behavior. Evidence limits: Linux and cross compilation do not run the macOS-only reaping/lifetime tests or qualify a VM. The portable owner-loop regression covers production decision dispatch, and native child tests cover reaping; replacing the dispatched effect body still needs native integration coverage. The terminal fixture retains a narrow setup window before its identity marker. Cross-provider recovery remains out of scope; latest provider pin is retained. Real-VM check: kill one publisher while the provider lease is busy, confirm ps marks its port down, restart healthy loopback and routed siblings without graph cleanup or dead-publisher resurrection, then stop via Ctrl-C/down; also repeat the reused-frontend-PID recovery and group/pty/terminal-loss lifecycle cases. The full local gate before the base integrations passed all 117 TypeScript files, default Rust library tests (934 passed, 37 ignored), feature library tests (955 passed, 46 ignored), relay tests (34), installed-candidate contracts (4), feature Clippy, macOS ARM64 feature cross-Clippy, typecheck, lint, staged leak scan and registered models. The portable replacement-readiness negative control fails under the old global-exit rule. The new macOS two-child test exercises real reaping with a substituted publication read; it does not exercise Engine or guest state. The one-service restart model does not cover cross-service readiness. Merged `origin/next` through 26a1429 in normal merge commit 830387d. Both model-coverage rows are retained; #729 setup and #732 log follow integrate alongside the lifetime changes. #737's validated detached handoff carries #739's provider-pin reason; #737 static preflight and #738 detached readiness remain intact. Static-start provenance comes only from the frontend review, survives safe wrappers, and is checked across IPC; runtime error output cannot mark a failure as static. Regressions cover the same refusal code before and after start, malformed phase metadata and a cold-pool source-CLI UDP refusal with no boot request. The same persistent reviewer is clean on the exact committed tree. Final focused integration run: 361 passed, 16 skipped across 11 files; typecheck, lint, privacy and staged leak scan passed. Hosted CI and Codex at this new head remain pending; #733 may select a test subset. Lead is running the equivalent local merge on Mac (ctrl-c, kill-publisher, hup, up-twice, up-partial, webdb, recover-old). <!-- codesmith:footer --> --- <a href="https://backend.blacksmith.sh/track/run-proof?expires=1797098296&installation_model_id=17053&pr_number=739&ref=codesmith_pr_footer&repository=hack-dance%2Fhack&return_to=https%3A%2F%2Fgithub.com%2Fhack-dance%2Fhack%2Fpull%2F739&variant=proof_this_pr&signature=56e5892802de151e0d054597bf28d494ae100896f4a525cc4afca4d038d47c51"><picture><source media="(prefers-color-scheme: dark)" srcset="https://pr-comments-assets.blacksmith.sh/codesmith/proof-proof-this-pr-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://pr-comments-assets.blacksmith.sh/codesmith/proof-proof-this-pr-light.svg"><img alt="Proof this PR" src="https://pr-comments-assets.blacksmith.sh/codesmith/proof-proof-this-pr-dark-3x.png" width="121" height="28"></picture></a> <sup><a href="https://app.blacksmith.sh/hack-dance/codesmith/hack/pr/739?autoLogin=true&ref=codesmith_pr_footer">Open in [code]smith</a> · Need help on this PR? Tag <code>@codesmith-bot</code> with what you need.</sup> <!-- codesmith:autofix:disabled --> <!-- /codesmith:footer --> --------- Co-authored-by: Kit <344842+roodboi@users.noreply.github.com>
1 parent 5e017fd commit 79ab31a

51 files changed

Lines changed: 1824 additions & 255 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎docs/guides/native-candidate.md‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -301,6 +301,9 @@ as UDP publication refuses without starting a cold pool. Admitted builds use
301301
review-only image placeholders for this check; startup still builds or selects
302302
the real image and repeats its review before graph admission. Missing or invalid
303303
static-review metadata refuses before boot and names the preflight command.
304+
Detached `up` identifies a static refusal as “nothing was started” and names the
305+
configuration fix and preflight command. Failures after startup begins retain
306+
the guidance to inspect the run and preserve its recovery evidence.
304307
The standalone preflight does not cover:
305308

306309
- A project with native project inputs: a native project file
@@ -4081,3 +4084,32 @@ A publisher crash has a different effect: the owner reports the exit and keeps
40814084
the services running, with that published port marked down in `ps` and `status`.
40824085
Run `hack down` to stop the project with data retained. A publication notice is
40834086
not evidence that cleanup completed, and no publisher is restarted automatically.
4087+
4088+
Publisher exit observation never waits for the provider lease in the owner's
4089+
event loop. If another operation holds that lease, `ps` already marks the port
4090+
down and the runtime degraded. The owner retains the unreaped child and retries
4091+
the identity-bound journal commit; it announces the exit only after that commit.
4092+
Ctrl-C and authenticated `down` remain available. Healthy sibling publications
4093+
keep following their admitted restart policies; an exited publisher's own service
4094+
generation cannot recreate that publisher or rearm its route probe.
4095+
If one service has several publications, losing one pauses restart following for
4096+
all that service's publications until an explicit restart; other services continue.
4097+
Replacement readiness checks that replacement's own child and publication, so a
4098+
retained sibling exit (pending or committed) does not stop unrelated services.
4099+
Fresh startup still requires every publisher live. A replacement port conflict
4100+
requires that replacement's committed exit and an occupied port; its own live
4101+
listener is never classified as a conflict because another publisher exited.
4102+
4103+
New macOS frontend finalization records include the process birth time. Retaining
4104+
recovery can distinguish a reused PID without signalling its replacement. A
4105+
matching zombie also counts as exited. Unreadable identities refuse; records
4106+
without a birth retain their PID-only behavior. Linux does not compare `ps`
4107+
births because clock corrections can change that observation. Failure to capture
4108+
the frontend's own birth falls back to the legacy behavior instead of failing up.
4109+
4110+
A pool started by a different provider version reports `provider_pin_changed`
4111+
through `ps --json`, detached `up` and `exec`, with the same original-release or
4112+
home-retirement guidance. This does not migrate the pool or permit cleanup by a
4113+
different provider version. When publisher identity cannot be confirmed, recovery
4114+
explains that it waits for the process holding that PID to exit and names
4115+
`hack down --recover` as the retaining retry.

‎docs/reference/native-publisher-stop.mmd‎

Lines changed: 13 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,17 @@ stateDiagram-v2
66
[*] --> ready
77
ready --> enrolled : Interrupt
88
ready --> enrolled : FrontendLost [lifetime watch]
9-
ready --> publisher_down : PublisherCrash [exact Child reaped]
9+
ready --> publisher_pending : PublisherCrash
10+
publisher_pending --> publisher_down : CommitExit [lease acquired]
11+
publisher_pending --> enrolled : Interrupt or FrontendLost
12+
publisher_pending --> intent : Recover [owner absent]
13+
state "publisher-exit-pending" as publisher_pending
14+
note left of publisher_pending
15+
Port down; services remain running
16+
Provider busy retains the zombie
17+
No committed proof or notice yet
18+
Retry without blocking stop control
19+
end note
1020
publisher_down --> enrolled : Interrupt or SIGHUP
1121
publisher_down --> enrolled : FrontendLost [lifetime watch]
1222
publisher_down --> intent : Recover [owner absent]
@@ -20,7 +30,8 @@ stateDiagram-v2
2030
ready --> intent : Recover [owner absent]
2131
enrolled --> intent : Recover [owner absent]
2232
intent --> intent : StopPublisher [identity matches]
23-
enrolled --> bridge_retired : ReleaseBridge [publisher absent]
33+
enrolled --> enrolled : CommitExit [lease acquired]
34+
enrolled --> bridge_retired : ReleaseBridge [exit proof committed]
2435
intent --> bridge_retired : ReleaseBridge [proof current]
2536
bridge_retired --> intent : Recover [owner absent]
2637
bridge_retired --> stopped : CleanupGuest [proof current]

‎docs/reference/native-publisher-stop.svg‎

Lines changed: 2 additions & 2 deletions
Loading

‎docs/reference/v5-model-coverage.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,7 @@ must be attached to the eventual exact integration revision.
3737
| Property and critical states | Model coverage | Implementation/regression boundary | Remaining qualification |
3838
| --- | --- | --- | --- |
3939
| Native log follower versus lifecycle mutation, frontend loss, stopped container and transport loss | No new state model: the follower observes without a mutation lease and grants no cleanup authority | `provider/graph/log_follow.rs` and its native counterpart pin Ready admission, container and start generation; `provider/engine/log_follow.rs` bounds framing and distinguishes real EOF from stopped-service idle cutoff. `lifecycle_log_follow_tests.rs` keeps the production transport on a fake Engine socket active while acquiring startup and operation leases, with a same-lock contention control and output while each lease is held. `native-log-follow-source.test.ts` guards the observer-only route; process tests cover cancellation, frontend loss, truncated/partial output and idle end reasons | 4 KiB envelopes, 8 KiB wire lines, 64 Ki-character retained line prefixes. The portable lease test fakes identity/state checks; it does not exercise host process/disk auditing, graph admission or the VM relay. Real relay EOF on stop/down/restart, VM loss and concurrent lifecycle latency require Apple Silicon acceptance. No performance or whole-product parity claim. |
40-
| Foreground group interrupt; publisher exit before authenticated stop; dead owner's pending cleanup enrollment and bridge release | native-publisher-stop | [Process adapter](../../src/backends/native-project-process.ts) isolates Compose owners regardless of hooks; Rust `process::publisher_command` isolates publishers; `legacy_cleanup_enrollment` binds the old coordinator and inventories before a distinct recovery intent. [Group regression](../../tests/native-foreground-group-interrupt.test.ts) signals the whole group for fresh/restore owners; Rust real-publisher process, frontend-loss, exact child-reaping and enrollment tests check grouping, lifetime wakeups and stale selectors. | One graph, one publisher, one crash, one identity replacement and one audited successor boot; 110 states. SIGHUP requests retaining cleanup; a reaped publisher enters an observable publisher-down state until explicit stop. Native process tests and frontend framing/ps tests cover signal delivery and notices. Dependency release is a separate post-completion step; `NoStrandedReservation` rejects success with a held reservation. Real reservation-file regressions check retry, capacity reuse and refusal fences. Separate stopped-receipt and intent-completion commits are collapsed; TypeScript dispatch and Rust completion-digest regressions cover that interruption window. Publication-registry sync/rename is abstracted and covered by portable commit-interruption tests. Kernel identity, socket/inode proofs, guest observations, durable storage and existing next.13 home recovery remain implementation/Apple Silicon VM gates. No eventual recovery is asserted. |
40+
| Foreground group interrupt; publisher exit before authenticated stop; dead owner's pending cleanup enrollment and bridge release | native-publisher-stop | [Process adapter](../../src/backends/native-project-process.ts) isolates Compose owners regardless of hooks; Rust `process::publisher_command` isolates publishers; `legacy_cleanup_enrollment` binds the old coordinator and inventories before a distinct recovery intent. [Group regression](../../tests/native-foreground-group-interrupt.test.ts) signals the whole group for fresh/restore owners; Rust real-publisher process, frontend-loss, exact child-reaping and enrollment tests check grouping, lifetime wakeups and stale selectors. | One graph, one publisher, one crash, one identity replacement and one audited successor boot; 168 states. Publisher exit observation is separate from lease-bound reaping/proof commitment; `NoUncommittedExit` rejects owner cleanup while proof is pending. A busy lease stutters without delaying stop control. SIGHUP requests retaining cleanup; a reaped publisher enters an observable publisher-down state until explicit stop. Native process tests and frontend framing/ps tests cover signal delivery and notices. Dependency release is a separate post-completion step; `NoStrandedReservation` rejects success with a held reservation. Real reservation-file regressions check retry, capacity reuse and refusal fences. Separate stopped-receipt and intent-completion commits are collapsed; TypeScript dispatch and Rust completion-digest regressions cover that interruption window. Publication-registry sync/rename is abstracted and covered by portable commit-interruption tests. Kernel identity, socket/inode proofs, guest observations, durable storage and existing next.13 home recovery remain implementation/Apple Silicon VM gates. No eventual recovery is asserted. |
4141
| Exclusive claims; competing writers; partial owner publication | graph-admission, registry-writer, publication-journal, missing-publication-lock, native-frontend-recovery | Admission/registry locks, pending publication and exact saved owner; related runtime ownership tests | Real filesystems, boot/incarnation identity and unsupported writers remain independent. |
4242
| Interrupted native restart: Ctrl-C or a frontend crash while the old run stops; `up` and `restart` refusing each other | native-restart-intent | The restart's stop requests run in their own process group (`processGroup: "own"` in [native-runtime-client](../../src/backends/native-runtime-client.ts)); a resumed `prepared` intent accepts a graph its owner stopped ([restart](../../src/backends/native-project-restart.ts), [preflight](../../src/backends/native-project-restart-preflight.ts)); [restart regressions](../../tests/native-project-restart.test.ts) send a real group SIGINT and cover the resume; HACK-2035 regressions force SIGINT/SIGTERM child rejection before frontend abort in preflight and down, using the runtime client’s preserved `stopSignal` | The finite model abstracts interrupt delivery as one action and does not prove child-exit versus frontend-handler ordering; deterministic tests cover that classification. A stale restart lock after a frontend crash; down hooks still share the terminal's group; physical guest acceptance. |
4343
| Foreground Compose startup failing after resources started; unconfirmed cleanup; unauthenticated final inspection | native-startup-owner | `recordFailedStartupOwner` in [native-project-start](../../src/backends/native-project-start.ts) publishes the authenticated failed run with a finalization lifetime; [start regressions](../../tests/native-project-start.test.ts) cover recording, an unauthenticated receipt and a retained mapping | A refused final inspection still leaves only the runtime journal; crash between finalization registration and mapping publication; physical guest acceptance. |
@@ -49,7 +49,7 @@ must be attached to the eventual exact integration revision.
4949
| Graph cleanup container stop batch: bounded in-flight stops, shared worker queue, connect-stage retry, batch-wide unreachable state, deadline expiry, worker panic | **Coverage gap:** no maintained model covers scheduling inside the stop batch; native-route-stop covers a different stop protocol | `dispatch` and `attempt` in [the stop batch](../../packages/runtime-core/src/provider/engine/stop.rs): at most 4 stops in flight, every worker joined; only `connect`/`connect_timeout` failures retry (25 ms doubling to 500 ms, within the batch deadline and 30 s of a stop's first connect failure); once one stop exhausts its window, later connect failures return at once; a deadline reached during backoff keeps the connect stage; `response`, `timeout` and `transport` never retry. Regressions in the same file's `provider::engine::stop` tests (bounded in-flight socket server, late socket, missing socket, 14-stop retry counts, persistent connect failures, oversleeping backoff, non-retried stages, worker panic) and `fourteen_service_connect_failures_keep_resumable_progress_and_retry_only_unstopped` | Tests cover exact attempt counts and the in-flight bound for fixed interleavings; no model explores all worker schedules or proves at most one delivered request per stop. Assumes a failed connect sent no request bytes and that the stop budget covers fully serialized grace periods. Graph-level resume after a partial batch is covered by tests only. Host qualification of the descriptor-pressure root cause (HACK-1302) remains open. |
5050
| Concurrent stop/publication; route claim and relay lifetime | native-route-stop, shared-https-lifetime, previous-boot-shared-https, relay-lifecycle-barrier, relay-authorization, relay-staging, relay-fence | Fenced route removal, relay owner, ready/removed protocol, authenticated observations | Native DNS/TLS/browser behavior; old-boot uncertainty and external routing. |
5151
| Shared host helper instance: overlapping ups, session adoption, stale create decisions, replacement before finalize, epoch-record failure, interrupted cleanup, concurrent down, displaced launches, failed down session stop | shared-helper-instance | `src/lib/project-shared-helpers.ts` (shutdown generation, `assertCurrent`, `SharedHelperSessionOwnership`); `src/lib/project-lifecycle-controller.ts` session decision, the locked creation transaction (revalidate, publish, create, environment, record, stop on failure, all under one lock), launches fenced by the lock and a token check, abort and signal cleanup through the same epoch decision (`runIfOwned`), an unknown exact-presence answer failing the inspection, and the locked conditional finalize; `stopSessionBeforeReleasingHelpers` (down, restart, native retirement keep holders when the session stop fails); `runMuxCommand` bounds every lifecycle and backend mux call so it settles before the lock is released, and reports a timeout as unknown (never absence); `tests/mux-command-timeout.test.ts`; `tests/project-lifecycle-controller-ownership.test.ts`, `tests/project-shared-helpers.test.ts`, `tests/native-project-down.test.ts`, e2e `shared-host-helpers` | Helper holder/rollback and claim protocol are not modeled (claim fixtures cover them); timeout settlement is an assumption the code does not guarantee (a killed tmux client's queued command may still apply, so a late kill or window command can reach a newer session; consequences and the session-ID fence, not in this slice, are in the README); zellij is refused for shared-helper lifecycle sessions (token written after creation); the gap between a down's session stop and its release (an up that captured after the stopping mark can still succeed there) is a documented first-slice limitation; real mux/process/lock-file behavior. |
52-
| Engine restart policies: stop versus pending restart, guest engine restart, stale incarnations, terminal proof | service-restart-policy | Receipt-admitted policy, disarm-before-stop and `restarting` stop preflight in `shutdown.rs`; admission refusals for incarnation-bound delivery classes in `restart_policy.rs`; owner-driven `Rebind` after full retirement, fresh ownership/attachment checks and a new reservation in `bridges/restart_rebind.rs`; route probe sample-generation rearm; Rust tests for admission, endpoint identity, port-busy refusal, disarm and stop preflight | Real guest engine behavior (crash restart, stop during backoff, disarm before deletion) needs native host evidence; several services, backoff timing and retry counting are not modeled. Same-port/route reachability after a policy restart, route probe rearm and per-slot child retirement still require real-VM host evidence. Rebind abstracts retirement/reservation/publication; relay lifecycle, fence and staging models cover their separate failure boundaries. |
52+
| Engine restart policies: stop versus pending restart, guest engine restart, stale incarnations, terminal proof | service-restart-policy | Receipt-admitted policy, disarm-before-stop and `restarting` stop preflight in `shutdown.rs`; admission refusals for incarnation-bound delivery classes in `restart_policy.rs`; owner-driven `Rebind` after full retirement, fresh ownership/attachment checks and a new reservation in `bridges/restart_rebind.rs`; route probe sample-generation rearm; Rust tests for admission, endpoint identity, port-busy refusal, disarm and stop preflight; portable selected-child readiness guard plus macOS two-real-child polling regression prevent historical sibling exits from rejecting a replacement or misclassifying its own listener | Real guest engine behavior (crash restart, stop during backoff, disarm before deletion) needs native host evidence; several services, selected-child readiness during sibling failure, backoff timing and retry counting are not modeled. Same-port/route reachability after a policy restart, route probe rearm and per-slot child retirement still require real-VM host evidence. Rebind abstracts retirement/reservation/publication; relay lifecycle, fence and staging models cover their separate failure boundaries. |
5353
| Foreground Ctrl-C down hooks: concurrent `down`, second interrupt, crash between or during hooks, stop during `up.after`, next generation | native-foreground-down-hooks | Value-free compare-and-swap journal in [down hooks](../../src/backends/native-project-down-hooks.ts), foreground orchestration in [foreground stop](../../src/backends/native-project-foreground-stop.ts) and `hack down` in [native down](../../src/backends/native-project-down.ts); [journal](../../tests/native-project-down-hooks.test.ts), [foreground](../../tests/native-project-start-stop-hooks.test.ts) and [down](../../tests/native-project-down-journal.test.ts) regressions | Durable writes are atomic in the model; real fsync/power loss, a stale journal lock after a crash inside the lock, and the live process-group behavior of the graph owner on a host are not covered. One generation and at most two `down` invocations are explored. |
5454
| Dependency rebind, automatic refresh, incomplete initializer, terminal reuse | dependency-rebind, dependency-auto-refresh, stopped-pool-startup, restore-history | Dependency claim/readiness, saved execution, restore history and source convergence tests | Actual installer failure, cache corruption, retained application state and resource plateau. |
5555
| Private environment allocation lifetime: staged values and private file members, crash between record and stage, retirement with the container | **Coverage gap:** no maintained model covers environment or credential delivery; file members share the existing allocation, intent record and retirement, so the state machine is unchanged | `packages/runtime-core/src/provider/environment.rs` (preparation bounds, staging layout, verification), `managed_environment.rs` (envelope v2 refusals, forwarding), `graph/launcher.rs` (read-only binds, overlap refusals) and `environment_recovery.rs` retirement tests | Live VM evidence that only the owning UID reads a member, writes fail read-only and the member is absent after cleanup; refresh or rotation of delivered files is not implemented. |
@@ -73,7 +73,7 @@ liveness/reachability controls have their own documented assumptions.
7373

7474
| Model | Positive distinct states | Configurations |
7575
| --- | ---: | ---: |
76-
| [native-publisher-stop](../../tests/models/tla/native-publisher-stop/PublisherStop.tla) | 110 | 6 |
76+
| [native-publisher-stop](../../tests/models/tla/native-publisher-stop/PublisherStop.tla) | 168 | 7 |
7777
| [shutdown-rebase](../../tests/models/tla/shutdown-rebase/ShutdownRebase.tla) | 43 | 3 |
7878
| [native-plan-reconcile](../../tests/models/tla/native-plan-reconcile/Reconcile.tla) | 747,656 | 4 |
7979
| [native-cache-reconcile](../../tests/models/tla/native-cache-reconcile/CacheReconcile.tla) | 2,208 | 5 |

0 commit comments

Comments
 (0)