Skip to content

release: 0.7.1 — the runtime check reads verbose output; silent failures need another conversation's run - #29

Merged
sanctrl merged 4 commits into
mainfrom
fix/runtime-check-071
Nov 20, 2026
Merged

sanctrl merged 4 commits into
mainfrom
fix/runtime-check-071

Conversation

@sanctrl

@sanctrl sanctrl commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

What changes

Patch release 0.7.1 (fixed existing behaviour → patch +1, VERSIONING.md). Fixes from an independent review of 0.7.0 and a second one of this PR, all to the could-not-process path, plus the Windows shell-run fix from #28:

  1. The Claude Code runtime check reads verbose output, and is judged by its shape (src/presets/claude-code.ts, claudeRuntimeCheckJudge, RUNTIME_CHECK_SETTINGS). With verbose on (--verbose, the user's verbose setting or its ~/.claude.json copy, viewMode: "verbose"), claude -p --output-format json prints a JSON array of every message, and the judge refused every array: the check never passed, so a lone failed conversation was never reported on it and kept being run again. The judge now reads the array's last type: "result" element; the check's --settings turn verbose off (verbose: false, viewMode: "default") and say language: "english". Ok only on subtype: "success", is_error: false, num_turns >= 1 and, when usage is there, usage.output_tokens >= 1; the answer's words are not read ("Okay" or an answer in another language is an answer).
  2. A silent failure counts as a timeout (src/integration/spawn-run.ts, src/integration/integration.ts). A run that ran past 5 s and then exited non-zero with nothing on stdout (a crash with no output) is RunOutcome.silent and reaches the kit as RunEnd.timedOut, so it is reported only once another conversation's run completed, never on the runtime check alone. A provider refusal prints API Error: … on stdout and stays a plain failure, and a run whose stderr says API Error is not marked silent either. The detail still says … with nothing on stdout (it ran, then ended). The 5 s window is measured to the process's exit, not to classification.
  3. The could-not-process log line counts hidden_count, not the at most 200 listed ids (falls back to the list on a server without it).
  4. The Codex runtime check (src/presets/codex.ts): ok only on exit 0 with a turn.completed and an agent_message with text in the --json events, and no turn.failed (codexRuntimeCheckJudge, bounded capture, never logged); model_reasoning_effort="low" dropped (Codex sends a configured effort as is, so a model without low refused every check). The user's hooks stay on, as in the runs: a hook that breaks every run must fail the check too, since a Codex run prints --json events and is never silent.
  5. doctor's runtime-check warning only while the status file is fresh (written within 3 min; a running daemon rewrites it every 30 s); otherwise doctor says "the daemon has not written its status for — it may be down", the words status uses. A paused conversation is still said either way.
  6. A shell run keeps its exit code on Windows (from fix(spawnRun): a shell command's exit 1 is a failed run on Windows, not ENOENT #28): spawnRun({ shell: true }) uses Node's own spawn, so a command line that ran and failed with 1 is failed, not spawn … ENOENT / not_started.

Version 0.7.1 in package.json and src/version.ts; CHANGELOG [0.7.1] — 2026-11-20; README updated (Could not process, the runtime check bullets, doctor, run classification).

Verification

  • Tests: pnpm type-check clean; pnpm build clean; vitest run 39 files, 646 passed, 3 skipped (platform-only), on Node 22.23.1 / macOS; CI on ubuntu, macOS and Windows. New tests: the Claude judge on an object, a verbose array, an error result, num_turns 0 or missing, usage.output_tokens 0 or missing, "Okay" and non-English answers passing, and the runner with verbose output forced; the Codex judge on real-shaped events and the runner on each; spawnRun's silent flag, the stderr API Error guard, a shell run past the window (failed + silent), and the exit-time window; a silent failure not reported on the runtime check and reported as timed_out with other_run_completed; hidden_count in the log; doctor on a stale and a fresh status file. The silent-failure, hidden_count, exit-time and stderr-guard tests were each run against the code without their change and fail there.
  • Review: independent review of 0.7.0 (the findings fixed here), then an independent review of this PR at 9c9f963: no blockers; its requested changes are in d181a79.
  • Live check: n/a: not released (no tag, no publish). Before release, against local stand-in model providers with throwaway config homes and fake keys (no real account): Claude Code 2.1.328 printed a JSON array with 0.7.0's check flags under the user's verbose: true, viewMode: "verbose" or ~/.claude.json verbose, and one result object with 0.7.1's; with the user's language: "japanese" the request carried the Japanese instruction without our key and the English one with it; hooks stayed off with the new keys, and an unknown key or invalid value in --settings left the rest in force; ClaudeRunner.checkRuntime from this build: ok with the user's verbose setting on, not ok on a provider refusal. Codex 0.154.0: the request carries no forced effort, and CodexRunner.checkRuntime from this build is ok on an answer and not ok on a refusal. A refused claude -p --output-format text run prints API Error: … on stdout (exit 1).

Risk and rollback

  • A result with no turn or no output tokens no longer passes the Claude check. That direction only withholds evidence: the failure waits for another conversation's run, as without a check.
  • A silent failure is no longer reported on the runtime check alone, so a lone conversation that fails silently waits for another conversation's run (or the server's re-wakes) before it is reported.
  • The Codex check now runs the user's hooks, as each run does; a hook with side effects fires once per check.
  • Rollback: revert this PR; 0.7.0 stays the published version until a tag is pushed.

…res need another conversation's run

Patch release (fixed existing behaviour → patch +1, VERSIONING.md).

- Claude Code runtime check: the judge reads verbose output (a JSON array,
  judged on its last type "result" element) and requires num_turns >= 1 and
  a result that answers OK; RUNTIME_CHECK_SETTINGS turns verbose off
  (verbose: false, viewMode: "default").
- A run that exits non-zero with nothing on stdout past the 5 s window is
  RunOutcome.silent and reaches the kit as RunEnd.timedOut: reported only
  once another conversation's run completed. The window is measured to the
  process's exit.
- The could-not-process log line counts hidden_count, not the capped id list.
- Codex runtime check: --disable hooks; ok only on exit 0 with a
  turn.completed and an agent_message in its --json events
  (codexRuntimeCheckJudge); no forced model_reasoning_effort.
- doctor says the runtime check failed only while the status file is fresh;
  otherwise it says the daemon is not running.
`spawnRun({ shell: true })` runs the CLI's `listen --run` command line.
A command that got going and exited 1 must be a failed run with exit 1;
one the shell cannot find (sh 127, cmd.exe 9009, nothing on stdout) is
not started. Expected to fail on Windows until spawnRun stops routing
shell runs through cross-spawn, whose Windows hook turns every exit 1
of a shell spawn into a `spawn … ENOENT` error.
Shell runs (the CLI's `listen --run` command line) went through
cross-spawn. With `shell: true` cross-spawn resolves nothing, but its
Windows hook still re-emits every exit 1 as `spawn <command> ENOENT`:
a command that ran and failed with 1 became `not_started`, said to be
missing, and was never a could-not-process failure. Shell runs now use
Node's own spawn and keep the exit code; the existing rule tells a
command cmd.exe cannot find (exit 1, nothing on stdout: not started)
from one that ran and failed (exit 1 after output: failed). Direct runs
keep cross-spawn for PATHEXT and .cmd shim resolution.

The test's Windows exit code for a command not found is 1, as the
previous commit's CI run on Windows showed (cmd.exe /c exits 1).
…stderr are not silent

From the independent review of #29:

- Claude check: `language: "english"` in RUNTIME_CHECK_SETTINGS; the judge
  no longer reads the answer's words (drops /\bok\b/i: "Okay" or an answer
  in another language is an answer) and requires subtype success,
  is_error false, num_turns >= 1 and, when usage is present,
  usage.output_tokens >= 1. RUNTIME_CHECK_ANSWER is gone (never released).
- Codex check: `hooks` is no longer disabled. A hook that breaks every run
  must fail the check too; Codex runs print --json events, so a broken run
  is never silent and only the check can catch it.
- spawnRun: a run whose redacted stderr tail says `API Error` is not marked
  silent (a refusal moved to stderr stays a plain failure).
- doctor, stale status file: "the daemon has not written its status for
  <N> — it may be down", the words status uses (integrationStatusStaleFor
  replaces integrationStatusWritten).
- Tests: "Okay" and a non-English answer pass, num_turns 0 and
  output_tokens 0 fail; the stderr guard; a shell run that exits non-zero
  with nothing on stdout past the window is failed + silent.
- CHANGELOG and README follow.
@sanctrl
sanctrl merged commit e855f37 into main Nov 20, 2026
9 of 10 checks passed
@sanctrl
sanctrl deleted the fix/runtime-check-071 branch November 20, 2026 02:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant