Skip to content

fix(native): isolate authored fixture failures and expose closed diagnostics - #813

Draft
roodboi wants to merge 10 commits into
nextfrom
fix/native-authored-test-admission
Draft

roodboi wants to merge 10 commits into
nextfrom
fix/native-authored-test-admission

Conversation

@roodboi

@roodboi roodboi commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Authored startup failures currently collapse lock inspection, filesystem and host-process errors into the same artifact refusal. This diagnostic draft preserves closed cause and timing metadata before startup wrapping can hide it, and removes the host-process fixture's poison latch so one failed case restores its own environment and later cases report their own results.

This is not Ready. The post-parallel macOS root cause is still unknown; this PR does not classify it as test interference or a shared product resource, and does not change admission or recovery authority.

  • Keep the existing public refusal sentence and retain only fixed phase/cause codes, inspection kind, bounded duration and allowlisted errno. Never forward raw messages, paths, arguments, environment values or child output.
  • Capture per-case mux timings, including the real operation sent through tmux stdin. Emit bounded diagnostic tails only for failed or unsettled cases.
  • Restore per-case env/umask, settle owned mux groups, abort cooperative overrun work before restoring the environment, and retain failed roots for diagnosis. Isolate mux staging from other tests.

Current head 2e971f66 includes a normal merge of next at c3b79bc0. Focused validation on that head: 120 passed / 60 macOS-only skipped / 0 failed across 15 files; typecheck passed. Earlier 04ba3f38 also passed check/privacy and CLI build, and independent review cleared its diagnostic scope. The requested Muse replacement could not start. Pre-push inherited false positives were triaged separately.

second-family review: pending, lead

The full local gate was stopped after baseline fixtures rejected directories created under the host's umask 002. An isolated A/B reproduced that refusal at 002 and passed at 022. No full-suite pass is claimed; a complete run under 022 remains owed before Ready.

Next evidence: run the original six-way selected batch on macOS, then rerun each of these files in its own Bun process immediately afterward, for five consecutive rounds:

  • tests/native-authored-host-processes.test.ts
  • tests/native-authored-project-start.test.ts
  • tests/native-authored-project-start-recovery.test.ts
  • tests/native-authored-project-start-lifecycle.test.ts

Use the first failed case's closed artifact diagnostics and mux timings to identify the actual cause, then add its regression and fix in this PR. The original first failure took about 210 ms, so slow executable assessment and command timeout are hypotheses, not findings.

Diagnostic limits: host_deadline includes cooperative fixture abort; only the fixture emits that typed deadline. Repeated artifact samples can be one error crossing several wrappers. A started mux sample with no finished sample can also mean the surrounding process deadline was already expired and no child was spawned.

Tracks HACK-2219.

@vercel

vercel Bot commented Nov 14, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
hack-docs Ready Ready Preview Nov 15, 2026 7:38am UTC

Request Review

@roodboi

roodboi commented Nov 14, 2026

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-15T02:41:00.898797Z 5e798e0 Manual request
🔒 Security Review ✅ Completed 2026-11-15T02:44:28.336571Z 5e798e0 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Bravo.

Reviewed commit: 04ba3f38a1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: 04ba3f38a1

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@roodboi

roodboi commented Nov 14, 2026

Copy link
Copy Markdown
Contributor Author

Diagnostic draft updated to 2e971f6606a0bd27bc3c592d64fb85ced3fd49b2 by a normal merge of next at c3b79bc0, without conflicts. Focused checks: 120 passed, 60 macOS-only skips, 0 failures; typecheck passes.

Still not Ready: the post-parallel Mac root cause is unknown, and the five-round reproduction remains the next evidence. Use this updated head for that run. Full local gate remains unqualified as described in the PR.

second-family review: pending, lead

The requested replacement Muse model could not start; earlier review reports are retained. No fallback reviewer was used.

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. More of your lovely PRs please.

Reviewed commit: 2e971f6606

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: 2e971f6606

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

…est-admission

# Conflicts:
#	src/mux/mux-backend.ts
#	tests/native-authored-host-processes.test.ts
@roodboi

roodboi commented Nov 14, 2026

Copy link
Copy Markdown
Contributor Author

Diagnostic head for the Mac reproduction: 15344d2cf211b90e436189917a4b063cf13f54a6.

This is a normal merge of next at 1f761ac56 into #813. The remote had advanced one commit beyond the frozen 5002cf87bb next.17 candidate; this head includes that candidate and #836's activity-logging fix. No rebase or force-push.

Resolved both conflicts by retaining the diagnostics and the incoming changes:

  • tests/native-authored-host-processes.test.ts keeps per-case ownership, cancellation, bounded failure diagnostics, isolated mux staging and env/umask restoration. It also keeps fix(lifecycle): retry retirement reads and honor second Ctrl-C (HACK-2184) #809's complete “cancellation with a missing launch birth still ends the owned native mux session” case, including the separate cleanup-freshness checks.
  • src/mux/mux-backend.ts keeps the scoped, closed operation timings and next's updated tmux contract comments.

Local merged-head verification: 122 passed / 60 macOS-only skipped / 0 failed across 15 focused files, including the #809 cancellation case; typecheck and check/privacy pass. These are Linux results, not Mac acceptance. This remains a diagnostic draft with no root-cause claim.

Please run the six-way selected Mac batch, then each of the four affected files separately immediately afterward, for five rounds, at this exact head. Send the first failed case's sanitized complete output per round, including the mux timings JSON and its artifacts array; closed artifact metadata is captured before startup's outer error wrapper can hide it. Keep the timings from the parallel phase and immediate reruns so we can distinguish first failure from fallout.

The repeated 15–25 minute recovery window is recorded as evidence for investigating a resource that outlives a test process. The identity of that resource is still unproven; no timeout, permission, liveness or recovery check was relaxed.

@roodboi

roodboi commented Nov 14, 2026

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. 🚀

Reviewed commit: 15344d2cf2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: 15344d2cf2

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@roodboi

roodboi commented Nov 14, 2026

Copy link
Copy Markdown
Contributor Author

The Mac reproduction head is already posted above: 15344d2cf211b90e436189917a4b063cf13f54a6, including next at 1f761ac56. Keeping it stable for the five rounds. Recorded batch environment and new-executable scan backlog as ruled out; the sharper failing window is under four minutes. Found the exact orphan name in tests/daemon-sessions.test.ts: its session-name validation calls the real create route with myproject123 (also my-project and my_project) on the default tmux socket, with no cleanup. A separate fixture-isolation fix is underway. That leak is confirmed; its role in the authored-start failure remains unproven. No existing default-server session was touched.

roodboi added a commit that referenced this pull request Nov 14, 2026
…#844)

The daemon session route tests created `myproject123`, `my-project` and
`my_project` on the developer's default tmux server and never stopped
them. The first name matches the orphan reported after a parallel test
batch.

Run these route tests against a fresh private socket instead. A fixture
PATH shim invokes the real tmux executable with that socket and an empty
config, clears inherited `TMUX`, and starts a fixed sleeping command so
no user shell startup files run. Each case stops only its private
server, verifies it is absent, and restores PATH even when the case
fails. Final cleanup removes the owned fixture. Successful name
validation now requires an actual 201 response; the alphanumeric case
verifies its session exists on the private socket.

Validation: 39 tests across `daemon-sessions`, `session-list-stop-tmux`,
`shell-timeout` and `tmux-backend` pass locally with real tmux.
Typecheck, check/privacy and changed-file lint pass. The same persistent
independent Opus reviewer is clean after two rounds, including all three
session-presence assertions and guarded PATH restoration. No macOS
result is claimed yet; no extra macOS CI dispatch was requested.

This fixes a confirmed test leak found during HACK-2219 investigation.
It does **not** establish that the leak causes the post-parallel
authored-start failures. The diagnostic #813 head remains separate and
stable for the requested five-round Mac reproduction. Existing sessions
on the default server are untouched.

<!-- codesmith:footer -->
---
<a
href="https://backend.blacksmith.sh/track/run-proof?expires=1797249449&installation_model_id=17053&pr_number=844&ref=codesmith_pr_footer&repository=hack-dance%2Fhack&return_to=https%3A%2F%2Fgithub.com%2Fhack-dance%2Fhack%2Fpull%2F844&variant=prove_it_works&signature=04e3cfe568498b639fc36e746cc635e282edb594fa521a30f3724ecb3db14393"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://pr-comments-assets.blacksmith.sh/codesmith/proof-prove-it-works-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://pr-comments-assets.blacksmith.sh/codesmith/proof-prove-it-works-light.svg"><img
alt="Prove it works"
src="https://pr-comments-assets.blacksmith.sh/codesmith/proof-prove-it-works-dark-3x.png"
width="129" height="28"></picture></a>
<sup><a
href="https://app.blacksmith.sh/hack-dance/codesmith/hack/pr/844?autoLogin=true&ref=codesmith_pr_footer">Open
in [code]smith</a> · Need help on this PR? Tag
<code>@codesmith-bot</code> with what you need.</sup>

<!-- codesmith:autofix:disabled -->
<!-- /codesmith:footer -->

---------

Co-authored-by: Kit <344842+roodboi@users.noreply.github.com>
@roodboi

roodboi commented Nov 14, 2026

Copy link
Copy Markdown
Contributor Author

Updated head for the five-round Mac reproduction: 5772b3af000014e89c32cad60937383c9376e487.

This normally merges the exact frozen next.17 candidate 8e7242db9466fe14f52b9bfb4513c02fa9d68bf4 into #813, without conflicts. The diagnostics, per-case ownership/cleanup and #809 cancellation regression remain intact. The merge also includes next's separately landed daemon-session fixture isolation; no extra leak fix is mixed in here.

Local verification: 83 passed, 60 macOS-only skips, 0 failures across 14 focused files; typecheck passes. This remains a diagnostic draft, not root-cause acceptance.

Please use this head for the six-way selected batch, followed immediately by each of the four affected files separately, five rounds. Send the first failed case's complete sanitized output per round, including the mux timings record and its artifacts array. The batch environment and system scan backlog remain ruled out. No deadline or ownership check was relaxed.

@roodboi

roodboi commented Nov 14, 2026

Copy link
Copy Markdown
Contributor Author

Lead Mac reproduction at 5772b3af00 (tree: next 8e7242d + this PR), rounds 1 and 2 of 5 (rounds 3 to 5 are still running; I add them when done). Apple Silicon, macOS 26, Bun 1.4.4. Each round: 508 selected test files six at a time, then at once each of the four files alone in its own Bun process.

round 1 parallel done 15:59:59: ok=502 fail=6 native-fails=4
  round 1 alone native-authored-host-processes FAIL(5) started=15:59:59 ended=16:00:00
  round 1 alone native-authored-project-start FAIL(5) started=16:00:00 ended=16:01:13
  round 1 alone native-authored-project-start-recovery FAIL(4) started=16:01:13 ended=16:02:04
  round 1 alone native-authored-project-start-lifecycle FAIL(1) started=16:02:04 ended=16:03:12
round 2 parallel done 16:34:27: ok=501 fail=7 native-fails=4
  round 2 alone native-authored-host-processes FAIL(5) started=16:34:27 ended=16:34:28
  round 2 alone native-authored-project-start FAIL(5) started=16:34:28 ended=16:35:41
  round 2 alone native-authored-project-start-recovery FAIL(4) started=16:35:41 ended=16:36:31
  round 2 alone native-authored-project-start-lifecycle FAIL(1) started=16:36:31 ended=16:37:38

So the failure reproduces every time: all four files fail inside the parallel phase and again alone, starting within one second of the parallel phase ending and lasting at least three minutes.

First failing case per round (tests/native-authored-host-processes.test.ts, "Source5 private process proof starts the shipping controller and exact-token stop preserves value privacy", about 200 ms; identical in both rounds apart from timings; home and temporary paths replaced):

Host process fixture mux timings: {"samples":[
  {"backend":"tmux","operation":"list-sessions","phase":"finished","elapsedMs":156,"timedOut":false,"exitCode":0},   // 225 ms in round 2
  {"backend":"tmux","operation":"has-session","phase":"finished","elapsedMs":12,"timedOut":false,"exitCode":1}],     // 15 ms in round 2
 "dropped":0,
 "artifacts":[{"phase":"action","inspection":null,"elapsed_ms":null,"system_code":null,"cause_code":"action_failed"}],
 "droppedArtifacts":0}
error: Native authored run artifact is unsafe, changed, or owned by another run; inspect native state before retrying. Values omitted.
 diagnostic: { phase: "action", inspection: null, elapsed_ms: null, system_code: null, cause_code: "action_failed" }
      at closedFailure (src/backends/native-authored-artifact-error.ts:71)
      at nativeAuthoredArtifactError (src/backends/native-authored-artifact-error.ts:101)
      at storage (src/backends/native-authored-project-run.ts:450)
      at async withNativeAuthoredProjectAdmission (src/backends/native-authored-project-run.ts:1016)
      at async tests/native-authored-host-processes.test.ts:176

What this shows and does not show:

  • Only two mux commands ran, list-sessions (exit 0, 150 to 225 ms) and has-session (exit 1, the session is absent). No new-session sample follows, so the action failed before or without creating the session, and it failed fast (the whole case takes about 200 ms), not by a timeout.
  • The diagnostic is the generic one: phase: "action", cause_code: "action_failed", no system_code, no inspection. The catch at native-authored-project-run.ts:447-450 turns any error that is not one of the two named classes into this value, so the real cause is still hidden. The next diagnostic needs the closed class of the error thrown inside the action (a fixed label per throw site or error class, an errno code when there is one), still without values.
  • tmux list-sessions taking 150 to 225 ms is slow next to tmux -V at about 10 ms on the same machine at the same time. Worth checking what the fixture's tmux server is doing at that moment (how many sessions it lists, whether it is a private socket shared across the parallel tests, and whether leftover sessions from the parallel phase are still being torn down).
  • In the other three files the failing cases take about 6 s each ("stage hook-up-before; no native consumer was started"), which is a deadline, so those may be a consequence of the same refusal retried until a deadline, or a second symptom.

Ruled out earlier on this Mac: the batch script's environment variables (the file passes with all of them set once the window has passed); a system scan backlog for new executables (first run of a fresh script costs a constant 130 to 250 ms here, the same before and after the parallel phase); leftover processes of the tests' own after the parallel phase (none found by name). The window closes by itself within about four minutes.

Ownership: the previous owner's account ran out of capacity; a takeover session now owns this PR and #826. I rerun on request: post the head in a comment.

🤖 Generated with Claude Code

A generic action failure reported only `action_failed`, so every plain
refusal in the admitted action looked the same. Diagnostics now also carry
the thrown value's built-in class, up to four Hack source frames relative to
`src/` (anchored to this checkout's source root, so no path prefix, message
or value leaves), and errno names from a wider closed set. Getters and proxy
traps still never run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@roodboi

roodboi commented Nov 14, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for rounds 1 and 2. New diagnostic head for one more Mac run: 9a8511353a937a611cba1ca084192af7b14730b1.

What the round 1 and 2 output narrows down. The fixture's tmux is the stand-in script, so the 150 to 225 ms list-sessions is that script starting, not a real tmux server. In the first failing case the order is: startMany resolves the mux (list-sessions), inspects the session (has-session, absent), then startProcess runs, in this order, the singleton decision (none here), prepareLaunch, then ensureSession, whose last step is new-session. The refusal is somewhere between has-session and new-session. In that span the only resource outside the test's own root is the launch delivery socket that prepareLaunch creates under the system /tmp (src/backends/native-authored-host-processes.ts, serveLifecycleProcessDelivery({ root: realpath("/tmp") })). That is a lead, not a finding yet.

What this head adds (src/backends/native-authored-artifact-error.ts; still value-free, getters and proxy traps never run):

  • sites: up to four Hack source frames of the thrown error, as lib/file.ts:line relative to src/, anchored to the checkout's own src/, so no path prefix leaves. This names the exact refuse() call site and its callers.
  • error_class: the nearest built-in class (Error, TypeError, …, non_error).
  • system_code now covers a wider closed errno set (EADDRINUSE, ENAMETOOLONG, ENOTSOCK, ETIMEDOUT, EINVAL, …). Before, any code outside the original ten fell into action_failed with a null code.

Ask: one round is enough, since it reproduces every time. Run the six-way batch, then the four files alone right after it, at this exact head. Please send the first failing case's artifacts entry (now with sites, error_class, system_code) from both the parallel phase and the alone run, plus one 6 s "hook-up-before" case. If it's cheap, during the window also send the entry count of /private/tmp (ls /private/tmp | wc -l) and df -i /private/tmp. Your HACK-2246 note about more than 1,100 hkgf-/hkro- owner directories there makes the shared /tmp worth ruling in or out.

Local (Linux) at this head: tests/native-authored-artifact-error.test.ts plus the three related fixture files, 56 pass, 0 fail; CLI typecheck, biome, privacy check pass. The four affected files are macOS-only here.

@roodboi

roodboi commented Nov 14, 2026

Copy link
Copy Markdown
Contributor Author

Lead Mac reproduction at 5772b3af00, all five rounds: the failure reproduced five times out of five, with identical counts each round.

round 1 parallel done 15:59:59: ok=502 fail=6 native-fails=4
  round 1 alone native-authored-host-processes FAIL(5) started=15:59:59 ended=16:00:00
  round 1 alone native-authored-project-start FAIL(5) started=16:00:00 ended=16:01:13
  round 1 alone native-authored-project-start-recovery FAIL(4) started=16:01:13 ended=16:02:04
  round 1 alone native-authored-project-start-lifecycle FAIL(1) started=16:02:04 ended=16:03:12
round 2 parallel done 16:34:27: ok=501 fail=7 native-fails=4
  round 2 alone native-authored-host-processes FAIL(5) started=16:34:27 ended=16:34:28
  round 2 alone native-authored-project-start FAIL(5) started=16:34:28 ended=16:35:41
  round 2 alone native-authored-project-start-recovery FAIL(4) started=16:35:41 ended=16:36:31
  round 2 alone native-authored-project-start-lifecycle FAIL(1) started=16:36:31 ended=16:37:38
round 3 parallel done 16:54:11: ok=502 fail=6 native-fails=4
  round 3 alone native-authored-host-processes FAIL(5) started=16:54:11 ended=16:54:12
  round 3 alone native-authored-project-start FAIL(5) started=16:54:12 ended=16:55:26
  round 3 alone native-authored-project-start-recovery FAIL(4) started=16:55:26 ended=16:56:17
  round 3 alone native-authored-project-start-lifecycle FAIL(1) started=16:56:17 ended=16:57:25
round 4 parallel done 17:15:22: ok=500 fail=8 native-fails=4
  round 4 alone native-authored-host-processes FAIL(5) started=17:15:22 ended=17:15:23
  round 4 alone native-authored-project-start FAIL(5) started=17:15:23 ended=17:16:38
  round 4 alone native-authored-project-start-recovery FAIL(4) started=17:16:38 ended=17:17:30
  round 4 alone native-authored-project-start-lifecycle FAIL(1) started=17:17:30 ended=17:18:39
round 5 parallel done 17:35:49: ok=500 fail=8 native-fails=4
  round 5 alone native-authored-host-processes FAIL(5) started=17:35:49 ended=17:36:00
  round 5 alone native-authored-project-start FAIL(5) started=17:36:00 ended=17:37:14
  round 5 alone native-authored-project-start-recovery FAIL(4) started=17:37:14 ended=17:38:04
  round 5 alone native-authored-project-start-lifecycle FAIL(1) started=17:38:04 ended=17:39:13
done 17:39:13

Every round: all four files fail inside the six-way phase and again alone, starting within one second of that phase ending: host-processes 5 of 6 cases (about 1 s for the file), project-start 5 cases, project-start-recovery 4, project-start-lifecycle 1. The first failing case and its diagnostics are the same as in the comment above in every round (tmux list-sessions exit 0, has-session exit 1, then phase: "action", cause_code: "action_failed", no system_code).

New timing fact: nine seconds after round 5's last alone run ended (three and a half minutes after its parallel phase ended), tests/native-authored-host-processes.test.ts passed 6 of 6 with the same environment variables. So the window is shorter than 3.5 minutes. A sixth round is running now that retries that file every 10 seconds from the end of the parallel phase until it passes and saves a full process list, the temporary-directory census and the count of open tmux sockets at the first failure and at the first pass; I post the flip time and the differences when it finishes.

🤖 Generated with Claude Code

@roodboi

roodboi commented Nov 14, 2026

Copy link
Copy Markdown
Contributor Author

Sixth round at 5772b3af00: the condition renews itself. This changes the picture, so read it before the earlier timing remarks.

After the parallel phase I ran tests/native-authored-host-processes.test.ts alone every 10 seconds:

parallel done 17:56:57 fail=8
try 1 at +11s rc=1  1 pass  5 fail 
try 2 at +32s rc=1  1 pass  5 fail 
try 3 at +43s rc=1  1 pass  5 fail 
try 10 at +130s rc=1  1 pass  5 fail 
try 20 at +271s rc=1  1 pass  5 fail 
try 30 at +392s rc=1  1 pass  5 fail 
try 36 at +458s rc=1  1 pass  5 fail 
... (36 tries, every one "1 pass 5 fail", up to +458 s; I stopped it there)
then: no test runs for 45 s -> the same file passes 6 of 6

So this is not a window that closes a few minutes after the parallel phase. Each failing run re-arms something that expires after more than about 10 s and at most about 50 s. Retried at a shorter interval it fails indefinitely; left alone for 45 s it passes. That also explains the earlier rounds: each file was run once, then left alone, and passed a few minutes later.

What it is not, from the snapshots taken at the first failure:

  • No tmux process existed on the host (the fixture's tmux is its own script; the 150 to 225 ms for list-sessions is this Mac's constant first-run cost for a freshly written script).
  • No leftover test process: nothing younger than ten minutes besides the runner itself and unrelated system processes.
  • Each run uses its own retained fixture root under the temporary directory, so whatever is shared between two independent Bun processes of this file is outside that root: a file, lock, lease or record at a fixed or identity-derived path (the Hack home, the shared temporary directory root, /private/tmp, a path derived from the fixture's fixed project identity), or a time-based check on such a path's age.

Where I would look: every path in the authored start, admission and host-process code that is not under the per-test root; anything compared against the clock with a bound between 10 and 60 seconds (a lease, a "recently active" guard, a staleness threshold, a retry budget); and what a refused run writes on its way out, since a refusal itself renews the condition. A closed diagnostic naming the throw site inside the action would settle it in one more run.

The first trigger is still the parallel phase (probably a test killed by its own timeout mid-start, leaving the same thing behind). After that, the failures feed themselves.

🤖 Generated with Claude Code

@roodboi

roodboi commented Nov 14, 2026

Copy link
Copy Markdown
Contributor Author

Thanks. The renewal result (fails indefinitely at a 10 s retry, passes after 45 s idle) narrows it well. One failing try at the diagnostic head should now name the throw site.

Ask: check out 9a8511353a937a611cba1ca084192af7b14730b1, run the six-way phase, then run tests/native-authored-host-processes.test.ts alone once within 10 s. Post the first failing case's artifacts entry. At this head it carries sites (up to four src/-relative file.ts:line frames of the thrown error), error_class, and system_code over a wider errno set. Before, codes such as EADDRINUSE, EINVAL or ENOTSOCK fell into action_failed with a null code, so the earlier output cannot rule them out.

What I checked on Linux at this head:

  • This file is not macOS-only. It runs here and passes 6/6.

  • Back-to-back runs pass. Killing a run with SIGKILL at 0.4 s, 0.8 s or 1.5 s, then rerunning at once, passes 6/6 every time. So a plain kill plus a quick rerun does not re-arm it on Linux. The renewing state is either macOS-specific or needs the parallel phase's real failure mode.

  • Between has-session and new-session, the code runs:

    • the owner checks (check(): admission lease, held directories, owner file; all inside the fixture root);
    • prepareLaunch, which creates the launch delivery socket under realpath("/tmp") (/private/tmp/host-launch-<random>/delivery.sock on macOS; random, never reused);
    • ensureSession, which writes the lifecycle state and checks freshness again.

    None of these is a 10–60 s clock check. The only lifetime-based mux sweep (hack-mux-env-*) uses a 10-minute TTL, which does not fit.

  • One hermeticity gap that does fit "outside the fixture root": the host-process fixture sets PATH, HACK_SESSIONS_MUX, HACK_TEST_MUX_STATE, TMPDIR and the umask, but not HOME. Anything on this path that resolves configuration or state from the home directory reads the real one. I have not found such a read in the failing span. Once sites names the location I will either fix it there or isolate HOME in the fixture, whichever the evidence shows.

@roodboi

roodboi commented Nov 15, 2026

Copy link
Copy Markdown
Contributor Author

HACK-2219, two more data points from the Mac batch of Nov 14–15 (tree = next 8e7242d + eight PR heads):

  1. Recent parallel load is not required. This batch's parallel phase effectively ended five hours before its "alone" phase (one unrelated test hung, see HACK-2247; nothing else ran in between). The four native-authored files still failed alone inside the batch script: native-authored-host-processes (first failure after 281 ms, then three at about 0.1 ms each), -project-start-recovery, -project-start-lifecycle, and -project-start ("pre-Ready host retirement requires settled authenticated removal", four cases at about 5.9 s each).
  2. By hand, 26 minutes later, same tree, same environment variables: all four files pass in sequence (6 s, 142 s, 145 s, 153 s). Running control-plane-extensions (the file the batch ran immediately before) and then native-authored-host-processes back to back by hand also passes, and no process is left alive between them.

So the failing condition follows the batch script's process context or what its parallel phase left on disk, not the clock: it survived five idle hours, and was gone for runs started from a fresh shell. Things that differ and are still unchecked: descriptors and limits inherited from the batch's xargs -P 6 parent, the process group and session of the batch, and files the parallel-phase runs of these same four files left outside their per-test roots. The batch logs for both phases are kept on the lead Mac for this tree; ask for any file's log.

A separate lead, not proven: macOS syspolicyd crashed four times in about 20 hours on this Mac, each time during test batches (HACK-2247). A child spawned in the same second never started. If a native-authored fixture's host process is spawned while the daemon is restarting, a start that never completes would also surface as a generic action_failed.

🤖 Generated with Claude Code

@roodboi

roodboi commented Nov 15, 2026

Copy link
Copy Markdown
Contributor Author

Thanks. "It follows the batch's process context, not the clock" fits all three comments. The 10 s retry loop was itself a script, and a fresh shell passed. So I stop treating it as something with a lifetime.

Linux, at 9a8511353 (none reproduces; all 6/6 pass):

  • stdin from /dev/null;
  • stdin closed;
  • setsid (new session, no controlling terminal);
  • an xargs -P 6 parent;
  • ulimit -n 256;
  • inherited ignored SIGINT/SIGQUIT, SIGTERM/SIGHUP, or SIGPIPE/SIGCHLD;
  • killing a run with SIGKILL partway through and rerunning at once.

So the trigger is macOS-specific in that context. The syspolicyd lead (HACK-2247) fits that. I cannot read Linear from this session, so I am working only from your summary here.

Please run this on the Mac, as two runs on one tree.

  1. Set up the diagnostic head in its own worktree, outside your batch tree (paths are placeholders):
cd <your hack checkout>
git fetch origin fix/native-authored-test-admission
git worktree add /tmp/hack-813 9a8511353a937a611cba1ca084192af7b14730b1
cd /tmp/hack-813 && bun install --frozen-lockfile
  1. Save this probe as /tmp/hack-813-probe.sh. It prints only closed facts (flags, counts, timings, env variable names), never values or paths:
#!/bin/bash
# usage: bash /tmp/hack-813-probe.sh <label> [home]
set -u
label=$1; mode=${2:-}
out=/tmp/hack-813-probe-$label-$(date +%H%M%S); mkdir -p "$out"
fresh_exec_ms() {  # cost of exec'ing a newly written shebang script
  local d; d=$(mktemp -d); printf '#!/bin/sh\nexit 0\n' > "$d/x"; chmod 700 "$d/x"
  local t0; t0=$(perl -MTime::HiRes=time -e 'printf "%d", time*1000')
  "$d/x"; local rc=$?
  local t1; t1=$(perl -MTime::HiRes=time -e 'printf "%d", time*1000')
  rm -rf "$d"; echo "$((t1-t0))ms rc=$rc"
}
{
  echo "label=$label mode=${mode:-default}"
  echo "stdin_tty=$([ -t 0 ] && echo yes || echo no) stdout_tty=$([ -t 1 ] && echo yes || echo no)"
  echo "ctty=$( (: </dev/tty) 2>/dev/null && echo yes || echo no)"
  ps -o pid=,pgid=,sess=,tpgid=,stat= -p $$ | awk '{print "own_pgroup="($1==$2)" tpgid_set="($4>0)" stat="$5}'
  echo "sigignore=$(ps -o sigignore= -p $$) sigmask=$(ps -o sigmask= -p $$)"
  echo "open_fds=$(ls /dev/fd | wc -l | tr -d ' ') nofile=$(ulimit -Sn)/$(ulimit -Hn) maxproc=$(ulimit -u)"
  echo "user_procs=$(ps -U "$(id -u)" -o pid= | wc -l | tr -d ' ')"
  echo "syspolicyd_pid_age=$(ps -o etime= -p "$(pgrep -x syspolicyd | head -1)" 2>/dev/null | tr -d ' ')"
  echo "fresh_exec=$(fresh_exec_ms) fresh_exec_again=$(fresh_exec_ms)"
} > "$out/context.txt"
env | cut -d= -f1 | sort > "$out/env-names.txt"
cd /tmp/hack-813
if [ "$mode" = home ]; then export HOME="$(mktemp -d)"; fi
HACK_DAEMON_AUTOSTART_DISABLE=1 bun test tests/native-authored-host-processes.test.ts > "$out/test.log" 2>&1
echo "rc=$? $(grep -E '^ [0-9]+ (pass|fail)' "$out/test.log" | tr '\n' ' ')" >> "$out/context.txt"
grep -m2 -E '"artifacts"' "$out/test.log" > "$out/artifacts.txt"
grep -m1 -A1 'diagnostic:' "$out/test.log" >> "$out/artifacts.txt"
echo "$out"
  1. Inside the batch. In your batch script, at the point where it runs the four files alone, add these two lines before them, so they run with the same parent, descriptors and limits:
bash /tmp/hack-813-probe.sh batch
bash /tmp/hack-813-probe.sh batchhome home
  1. From a fresh shell, after the batch ends:
bash /tmp/hack-813-probe.sh fresh
diff /tmp/hack-813-probe-batch-*/env-names.txt /tmp/hack-813-probe-fresh-*/env-names.txt
  1. Around the batch run, for HACK-2247:
log show --last 2h --style compact --predicate 'process == "syspolicyd" AND messageType == fault' | tail -20
ls /Library/Logs/DiagnosticReports | grep -c syspolicyd

Please post:

  • all three context.txt files;
  • the artifacts.txt files (the first failing case's sites, error_class and system_code name the throw site);
  • the env-names diff;
  • the fault count.

How I will read it:

  • If batch fails and fresh passes, the context lines show what differs: terminal, process group, ignored signals, descriptors, limits, or fresh_exec time.
  • If batchhome passes, the cause is state read from the real home directory. I fix the fixture's isolation and the code that reads it.
  • If fresh_exec in the batch is slow or fails, the syspolicyd lead is confirmed.
  • sites gives the exact line either way. The fix and its regression follow from that.

@roodboi

roodboi commented Nov 15, 2026

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3255a0ccb7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: 3255a0ccb7

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

… and forged sites

- A proxy error, or one with a proxy anywhere on its prototype chain, is
  refused before any reflection or instanceof, so none of its traps runs.
- Inspection fields and codes are read once as own data properties; a
  subclass getter is never invoked.
- `sites` accepts only frame lines (`    at …`) whose file is in a fixed
  inventory of this checkout's `src/` files, so message text or a forged stack
  cannot pass as a site.
- The stack and each frame line are size-bounded before matching; an
  oversized single-line stack no longer scans quadratically.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@roodboi

roodboi commented Nov 15, 2026

Copy link
Copy Markdown
Contributor Author

Status after next moved, not a ready note. This PR stays a draft until the Mac evidence names the root cause and the fix and its regression land here.

  • Normal merge of next at e47681ea0af4a332135cede4aa32f53e6fa7df27, no conflicts (3255a0ccb).
  • Review loop on this unit: one persistent Codex reviewer (codex-dio, gpt-6.1-sol).
    • Round 1 was NOT CLEAN with three findings in the new diagnostics:
      • sites could pass message text or a forged stack off as a source location;
      • classification could still run proxy traps and subclass getters;
      • a very long stack line made the frame regex scan quadratically (56 s on a 200k-character line).
    • All three are fixed in 5e798e0a7173b17a7fb019219d06b8f128cac6de. Each has a regression test that fails on the old code.
    • Round 2: CLEAN at 5e798e0. The reviewer's caveat: sites are diagnostic hints, not authenticated throw-site evidence. A forged stack could still name a real source file and a line number.
  • Ran at this head (Linux): CLI typecheck, biome, privacy check; 70 tests across the six focused files; bun run build; bun run check:native-plan-cli; bun scripts/check-config-compiler-cli.ts. All pass.

For the Mac probe in my previous comment, please use this head instead of 9a8511353. Only step 1 changes:

git fetch origin fix/native-authored-test-admission
git worktree add /tmp/hack-813 5e798e0a7173b17a7fb019219d06b8f128cac6de
cd /tmp/hack-813 && bun install --frozen-lockfile

Everything else in the probe stays the same.

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Hooray!

Reviewed commit: 5e798e0a71

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: 5e798e0a71

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@roodboi

roodboi commented Nov 15, 2026

Copy link
Copy Markdown
Contributor Author

Status after next bf74aaf. Still not a ready note. This PR's job is the HACK-2219 root cause plus its regression. That still waits on the Mac batch-versus-fresh-shell probe.

Mac probe: please use this head. Step 1 of the probe becomes:

git fetch origin fix/native-authored-test-admission
git worktree add /tmp/hack-813 2c9a55f37931a4a254d99c3f2a9ff47907487b45
cd /tmp/hack-813 && bun install --frozen-lockfile

The rest of the probe is unchanged.

Option for your merge sets: the diagnostics and per-case fixture isolation are reviewed and could land on their own, ahead of the root cause. If you want that, I will retitle this PR as diagnostics-only, mark it ready, and put the fix in a follow-up PR. My default is to keep it a draft until the probe names the cause, as the original brief asked.

@roodboi

roodboi commented Nov 15, 2026

Copy link
Copy Markdown
Contributor Author

Status after next 345dc0d. Still not a ready note, for the same reason as before: the HACK-2219 root cause waits on the Mac probe.

  • Normal merge of next at 345dc0d119495d5ec76470b32c14ee21fe3c996d, no conflicts. Head is now 0a8ffe02d71108344deb40335f035bc0dc2f7b61. The merge changed none of this PR's 13 files. The PR adds or changes no --json command and no generated file, so nothing was regenerated.
  • Ran at this head (Linux):
    • tests/machine-output-contract.test.ts, tests/sync-spawn-bound.test.ts and the six focused files: 159 pass.
    • CLI typecheck clean. Biome on this PR's files: only the complexity warning next already has. Privacy check ok.
    • bun run build and build:config-compiler.
    • The nine acceptance scripts pass (check:native-plan-cli, check-config-compiler-cli, check-test-config-compiler).
  • Reviewer (same Codex thread), round 4: CLEAN at 0a8ffe0. feat(projects): share backend-aware service state across CLI and daemon #831's shared service state and fix(cli)!: standardize HACK-2218 machine output contracts #835's envelope rule do not touch authored admission, host-process ownership, shell timeouts, mux execution or diagnostic redaction. Bot threads: CLEAR.

Mac probe: please use 0a8ffe02d71108344deb40335f035bc0dc2f7b61 in step 1 (git worktree add /tmp/hack-813 0a8ffe02d71108344deb40335f035bc0dc2f7b61); the rest is unchanged. The option to land the reviewed diagnostics first, as a diagnostics-only PR, still stands if you want it in a merge set.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant