fix(lifecycle): retry retirement reads and honor second Ctrl-C (HACK-2184)

docs/diagrams/compose-lifecycle-retirement.mmd

@@ -7,8 +7,10 @@ stateDiagram-v2
   attached --> captured : Compose exits
   captured --> signalled : owned mux stop
   signalled --> commandsRetired : no live command members
-  signalled --> retained : observation unknown
+  signalled --> signalled : failed census / retry within budget
+  signalled --> retained : no census succeeds by deadline
   signalled --> escalated : 5 s grace expires
+  signalled --> escalated : second Ctrl-C / identity rechecked
   escalated --> commandsRetired : no live command members
   escalated --> retained : deadline or probe error
   commandsRetired --> wrappersRetired : no live wrapper members
@@ -27,6 +29,8 @@ stateDiagram-v2
     FIFO EOF is not group absence
     TERM separately grouped descendants
     Revalidate member identity before TERM or KILL
+    Probe EPERM needs census
+    Delivery EPERM plus live members refuses
     Zombie-only groups are retired
     After KILL allow 2 s to confirm retirement
   end note

docs/lifecycle.md

@@ -28,16 +28,65 @@ again before signalling. The wrapper forwards TERM to its main command group;
 Hack sends TERM to separately grouped descendants that the wrapper does not signal.
 It waits for the detached commands to disappear before
 retiring the FIFO wrapper and its reader. Each phase allows 5 seconds for graceful
-exit, escalates surviving groups once with SIGKILL, then waits up to 2 seconds for
-retirement (at most 14 seconds of group waiting across both phases, plus bounded mux
-and process-identity queries). The capture allows two 1-second group snapshots
-alongside a 2-second identity read; each signal batch checks identities again with
-bounded queries. Explicit `hack down` uses the same command-before-wrapper order
+exit, escalates surviving groups once with SIGKILL, then gives confirmation a
+2-second polling budget. Each `ps` call can itself take up to 2 seconds, including
+a final call begun at the polling deadline: these budgets are not hard wall-clock
+limits. A failed census is retried within the current observation budget. If no
+read succeeds, cleanup stays unconfirmed; a successful live observation still
+requires retirement or escalation. Probe EPERM means the group may still exist: on
+macOS, even a zombie-only group can return it. Hack uses the process census to
+distinguish live members from zombies. TERM or KILL denied with EPERM is benign
+only when the following observation proves retirement; live members still cause
+a refusal. Failed reads retry within the existing phase budget.
+Explicit `hack down` uses the same command-before-wrapper order
 after interrupting its owned lifecycle panes.
 Commands that finish their TERM handler sooner do not wait out the grace period.
+A second Ctrl-C skips the remaining command grace and requests the single SIGKILL;
+identity/ownership checks and post-KILL confirmation still run. An in-flight
+bounded `ps` or mux call finishes before that request can take effect.
+The foreground guard counts Ctrl-Cs from before Compose starts: two interrupts
+while Compose stops containers also skip the later lifecycle command grace.
+The wrapper/reader phase keeps its five-second reap window even after a forced
+command stop, so the wrapper can remove its private FIFO directory before exiting.
+
+| Lifecycle shutdown case | With prompt process reads (excluding Compose and mux shutdown) |
+| --- | --- |
+| Command finishes a one-second TERM handler; wrapper reaps it | About 1 second, plus polling/scheduling |
+| Command ignores TERM; wrapper reaps after KILL | About 5 seconds, plus KILL confirmation/polling |
+| Command needs KILL; wrapper then also needs KILL | About 10 seconds, plus both KILL confirmations; confirmation polling budgets can add up to 4 seconds |
+| Every process observation consumes its timeout | Up to 48 seconds across both group phases, as detailed below; capture and mux checks are additional |
+
+The conservative 48-second group ceiling includes two initial-observation budgets
+of 5 seconds (up to 7 seconds each with one final 2-second read), two grace budgets
+(up to 7 seconds each), two confirmation budgets (up to 4 seconds each), and three
+signal-identity batches (up to 4 seconds each with one failed-read retry). No census
+success within an initial budget refuses cleanup without signalling. Capture adds
+up to 4 seconds: two 1-second snapshots run alongside up to two 2-second identity
+reads. The normal owned foreground mux-stop path uses a token read and a stop,
+each bounded by `HACK_MUX_COMMAND_TIMEOUT_MS` (15 seconds by default); a failed stop
+can add one bounded presence check. Thus the worst-case process/mux allowance is
+97 seconds before optional identity refresh, rather than 14 seconds. A missing
+launch birth adds up to two owner reads, one parallel pane-query batch and two
+identity reads: at most another 49 seconds with default mux timeouts. These are
+conservative allowances for successful observations near deadlines, not a timer
+that every shutdown waits out. Filesystem I/O, OS scheduling, the instance-lock
+wait and Compose's own service stop grace are separate. Explicit `down` also sends
+an interrupt to each lifecycle window and waits 750 ms before this group path;
+its per-window mux calls and subsequent shutdown work are additional.
 An unreaped zombie has exited and runs no code: a group containing only zombies
 counts as retired. A live member in the same group still requires shutdown;
-failed process observation remains unconfirmed.
+failed process observations never count as absence.
+Native launch cancellation still permits the owned cleanup to retry a missing
+start time: its ownership and artifact checks remain active independently of the
+canceled startup deadline. Failed native launches also use a cleanup freshness
+check independent of canceled startup inputs: admission, hook ownership, private
+directory identity, the saved owner record and the mux token are still checked.
+Normal startup and ordinary stop retain their full input freshness checks.
+
+For value-free retirement diagnostics, set `HACK_LIFECYCLE_RETIREMENT_TRACE=1`.
+Stderr records fixed decision labels, PIDs/group IDs, counts and boolean outcomes.
+It excludes paths, names, command/env values, ownership tokens and raw errors.
+Leave it unset for normal operation.
 `hack doctor --fix` also captures groups before mux teardown and waits for
 commands before wrappers when repairing an ownership-proven orphan. An
 unconfirmed retirement keeps that entry and is reported while repair continues
@@ -56,8 +105,13 @@ grant signal authority or make an unrelated live group count as retained work.
 
 Zellij does not await process-group retirement; it is scheduled for removal in v5.0.0.
 
-Retrying missing launch-time identity observations during owned foreground cleanup
-remains a follow-up: a missing start time can currently leave cleanup unconfirmed.
+If the launch missed a pane start time, foreground cleanup re-reads the current
+pane in the recorded window while checking the session's ownership token before
+and after the query. It fills only a missing birth for that same current pane;
+it never replaces a stored birth or borrows identity from a reused PID outside
+the window. A failed pane query leaves the original missing birth intact: Hack
+still attempts the token-checked mux stop, then retains unconfirmed descendant
+state. A replaced owner still refuses the stop.
 
 ![Foreground lifecycle retirement](diagrams/compose-lifecycle-retirement.svg)
 

docs/reference/v5-model-coverage.md

@@ -67,7 +67,7 @@ must be attached to the eventual exact integration revision.
 | Post-Compose bridge attachment: endpoint placeholder persists or resolves to owned/foreign ID (HACK-2141) | **Coverage gap:** no maintained model of external Docker endpoint attachment timing | `waitForNativeComposeNetworkMembers` retries only owner-issued member-ID refusal, re-acquiring complete proof; `native-compose-ownership.test.ts` covers the three-ID/one-placeholder listing, resolution, persistence, foreign member, cancellation, deadline and nonretryable errors. `native-compose-endpoint-settlement-command.test.ts` drives source CLI `up` through readiness and the prepared guard independently, with a placeholder that resolves on the next query; strict/saved policy is unchanged | Two-second monotonic retry window plus one bounded probe; Docker read consistency and eventual attachment remain external assumptions. `native-config-process-policy` is the live isolated regression; repeated CI runs remain separate evidence. Other effect-adjacent reads still fail closed immediately: the routed-run guard, strict `afterReadiness` re-read, one-off run pre/post-removal checks, saved exec guards and other saved non-observe commands, saved stop/down reads, and file-owner pre/post-effect/readiness checks in `native-compose-file-command.ts`. A detaching-endpoint placeholder was not observed in this investigation; those paths remain known gaps. |
 | Compose detached startup: short job exits before observation; old completed container replaced between samples | **Coverage gap:** no maintained model of external Docker observation consistency | `waitForComposeStartupState` corroborates identities and classifies only the latest snapshot within the remaining start budget. `compose-startup-state.test.ts` covers sub-second interleavings, replacement, failure and deadline; `project-up-startup-state.test.ts` replays the Compose/Docker boundary. `compose-short-oneshot` checks real repeated replacement and sub-second exit zero | Assumes Docker listings eventually describe current containers; two matching identity listings cannot prevent replacement after the final read. Unit bounds are one job, two identities and a 1 s virtual budget. Real-project acceptance and Docker CI remain separate gates. |
 | Compose foreground first/second interrupt and lifecycle signal handoff | **Coverage gap:** no maintained model covers the external Compose shutdown protocol | `composeRuntimeBackend.up` isolates the Compose session, forwards each signal once, and waits for child exit/log EOF; `runUpCommand` retires the setup signal owner before attached launch. `tests/compose-foreground-signal.test.ts` covers delayed graceful completion, second interrupt and listener disposal; `tests/e2e/scenarios/compose-foreground-signal.ts` checks both real containers immediately after a process-group SIGINT and CLI exit. | Finite subprocess tests cover one or two interrupts and one child; Docker shutdown success is observed by the live regression, not inferred from child exit. Docker daemon failure, wrapper SIGKILL/crash and an unresponsive Compose client are not qualified. The first interrupt has no forced deadline; Compose owns its configured service stop grace periods. |
-| HACK-2184: foreground lifecycle process groups retire before successful cleanup/state removal | **Coverage gap:** no maintained model covers mux acknowledgement versus external group exit | `abortOwnedSession` awaits `createLifecycleSessionRetirement`; capture owned pane descendants before mux teardown, settle command groups before wrapper/reader groups, then remove owned state. `terminateLifecycleProcessGroups` observes absence after TERM and after its one KILL escalation. `tests/lifecycle-group-settlement.test.ts` uses deterministic time/signal boundaries; `tests/project-lifecycle-controller-ownership.test.ts` blocks retirement and verifies cleanup/state cannot complete early. Removing either wait boundary fails its regression. `tests/lifecycle-fifo-retirement.test.ts` exercises real delayed and TERM-resistant FIFO writers plus a separately grouped cooperative worker; only groups outside the wrapper’s forwarded TERM receive an additional identity-checked TERM; `repairProjectLifecycleSessions` uses the same ordered capture/retirement before removing an orphan entry, and `tests/lifecycle-hygiene-retirement.test.ts` gates state removal, retains uncertainty, continues other repairs and exercises a real TERM-resistant FIFO writer; the Docker foreground scenario requires separate app/worker ready markers, polls both named running containers before Ctrl-C, and verifies the one-second TERM handler finished without KILL. `tests/compose-foreground-readiness.test.ts` rejects replayed single-service markers and stale container listings; settlement, absent-session and repair tests cover zombie-only retirement and mixed live members. [Statechart](../diagrams/compose-lifecycle-retirement.svg), [source](../diagrams/compose-lifecycle-retirement.mmd). | Each group phase has 5 s grace plus 2 s confirmation after at most one KILL; two phases total at most 14 s plus bounded mux calls/process observations. ESRCH or a successful census containing only zombie members proves retirement; probe/permission errors retain uncertainty, ownership and dependent helpers. Deterministic cases cover selective TERM for two groups; real FIFO fixtures cover a main command waiting for a separate worker group, and the Docker fixture covers two services and one persistent command. OS scheduling, PID reuse outside the active ownership interval, CLI SIGKILL/crash, escaped descendants and zellij retirement are not qualified by these tests. HACK-2184 retains the follow-up to refresh missing launch-time births through owned cleanup. Zellij tracking is dropped because HACK-2215 replaces the lifecycle wrapper and zellij is scheduled for removal in v5.0.0. `up.after` after successful attached Compose return and explicit down hooks are inherited v4.3.0 ordering, documented in lifecycle.md. |
+| HACK-2184: foreground lifecycle process groups retire before successful cleanup/state removal | **Coverage gap:** no maintained model covers mux acknowledgement versus external group exit | `abortOwnedSession` awaits `createLifecycleSessionRetirement`; capture owned pane descendants before mux teardown, settle command groups before wrapper/reader groups, then remove owned state. `terminateLifecycleProcessGroups` observes absence after TERM and after its one KILL escalation. `tests/lifecycle-group-settlement.test.ts` uses deterministic time/signal boundaries; `tests/project-lifecycle-controller-ownership.test.ts` blocks retirement and verifies cleanup/state cannot complete early. Removing either wait boundary fails its regression. `tests/lifecycle-fifo-retirement.test.ts` exercises real delayed and TERM-resistant FIFO writers plus a separately grouped cooperative worker; only groups outside the wrapper’s forwarded TERM receive an additional identity-checked TERM; `repairProjectLifecycleSessions` uses the same ordered capture/retirement before removing an orphan entry, and `tests/lifecycle-hygiene-retirement.test.ts` gates state removal, retains uncertainty, continues other repairs and exercises a real TERM-resistant FIFO writer; the Docker foreground scenario requires separate app/worker ready markers, polls both named running containers before Ctrl-C, and verifies the one-second TERM handler finished without KILL. `tests/compose-foreground-readiness.test.ts` rejects replayed single-service markers and stale container listings; settlement, absent-session and repair tests cover zombie-only retirement and mixed live members. [Statechart](../diagrams/compose-lifecycle-retirement.svg), [source](../diagrams/compose-lifecycle-retirement.mmd). | Each group phase has a 5 s initial-observation retry budget, 5 s grace and 2 s confirmation after one KILL; each census can overshoot by 2 s. Two phases allow up to 48 s including bounded signal-identity retries, plus capture/mux/optional birth refresh (see lifecycle.md timing table). Second Ctrl-C skips command grace, not fences, exit confirmation or the wrapper reap window; the foreground guard also counts interrupts during Compose shutdown. Failed censuses retry within their deadline; no successful census remains unknown. ESRCH or a successful census containing only zombie members proves retirement; probe EPERM falls through to the zombie-aware census (including macOS zombie-only groups). Delivery EPERM is benign only after observed retirement; a live member refuses, and failed reads retry within the same phase deadline. Other probe errors retain uncertainty, ownership and dependent helpers. Deterministic EPERM-to-ESRCH, mixed-live and denied TERM/KILL cases plus a real held-zombie fixture on macOS/Linux cover these branches; Linux execution does not qualify macOS. Deterministic cases cover selective TERM for two groups; real FIFO fixtures cover a main command waiting for a separate worker group, and the Docker fixture covers two services and one persistent command. OS scheduling, PID reuse outside the active ownership interval, CLI SIGKILL/crash, escaped descendants and zellij retirement are not qualified by these tests. Native launch abandonment excludes canceled startup inputs and budget from missing-birth refresh, while rechecking admission, hook ownership, directory pins, saved owner and mux token; normal start/stop retains full input freshness. A portable adapter regression cancels both startup-input authority and budget; a real adapter regression holds and fails the launch census, cancels, then requires owned mux and state removal. The native recovery fixture retries probe EPERM until ESRCH instead of mistaking the error for absence. FIFO and held-zombie files run in both hosted OS pools. Owned foreground cleanup refreshes only missing births through current pane membership with owner-token checks on both sides; stored births and replacement refusals remain strict. tests/lifecycle-cleanup-followup.test.ts and the controller launch-read failure regression cover refresh, second-interrupt propagation and shared-helper fences. Zellij tracking is dropped because HACK-2215 replaces the lifecycle wrapper and zellij is scheduled for removal in v5.0.0. `up.after` after successful attached Compose return and explicit down hooks are inherited v4.3.0 ordering, documented in lifecycle.md. |
 
 These rows identify coverage, not closure. A local mutant or spy proves handling,
 not that the kernel/provider produces the modeled condition. No model establishes

docs/versions/v5/getting-started.md

@@ -99,8 +99,13 @@ Do not paste raw logs into a public issue.
 For foreground `hack up`, Ctrl-C stops the project instead. With tmux, Hack also
 waits for lifecycle commands and separately grouped workers: they get TERM and
 five seconds to finish handling it
-before one forced stop. Hack reports unconfirmed cleanup; repeated Ctrl-C does
-not skip that wait. Ctrl-C does not run `down.before` or `down.after` hooks;
+before one forced stop. A second Ctrl-C skips the remaining command grace; Hack
+still checks ownership, confirms exit and lets the wrapper finish cleanup.
+Slow process reads can extend shutdown;
+Hack retries transient read failures and reports cleanup it cannot confirm.
+Exited processes waiting to be reaped count as stopped, including on macOS.
+Canceling startup still allows cleanup to retry a missed process start time.
+Ctrl-C does not run `down.before` or `down.after` hooks;
 use explicit `hack down` for those hooks. Attached `up.after` runs only after
 Compose returns successfully, so it does not run while the project stays attached.
 

scripts/test-shard-plan.ts

@@ -514,6 +514,10 @@ export const MACOS_POOL_FILES: Readonly<Record<string, string>> = {};
  * added to the Linux pool as well (HACK-2142).
  */
 export const BOTH_POOL_FILES: Readonly<Record<string, string>> = {
+  "tests/lifecycle-fifo-retirement.test.ts":
+    "FIFO retirement needs Linux process groups and macOS zombie-probe EPERM coverage",
+  "tests/lifecycle-zombie-retirement.test.ts":
+    "held zombies return probe success on Linux and can return EPERM on macOS",
   "tests/v4-upgrade.test.ts":
     "historical upgrade must preserve keys and values on Linux and macOS",
   "tests/upgrade-installer.test.ts":