Repository navigation
Harness: sidecar verify, usage split, judge resilience, tool errors - #45
Conversation
Make verifyLogMode sidecar the default so full verify output lands under verify-logs/ while prompts get a tail-heavy preview plus path. Inline mode remains opt-in. In-memory capture uses head+tail with a middle elision marker instead of head-only truncation. Co-authored-by: Paul Zgordan <dancingteeth@users.noreply.github.com>
Sum uncached input, output, and cache read/write across worker and judge sessions. Surface totals in the usage log line and run-report billing rows. Cursor runs mark cache splits as unknown rather than zero. Co-authored-by: Paul Zgordan <dancingteeth@users.noreply.github.com>
Classify judge errors, retry the primary once, then try reviewSecondary. When both fail, log review_judge_failed, set status waiting, and exit 0 instead of treating the run as a hard verify failure. Co-authored-by: Paul Zgordan <dancingteeth@users.noreply.github.com>
Track tool_end failures and plain-text tool-call stalls on worker and judge sessions. Merge per iteration into log.ndjson and aggregate in run-report.md. Co-authored-by: Paul Zgordan <dancingteeth@users.noreply.github.com>
Co-authored-by: Paul Zgordan <dancingteeth@users.noreply.github.com>
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configuration
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Waiting status (budget, env, judge down, agent transient cap) exits 3 in the CLI and AGENT_LOOP_DONE. Document exit codes in README and CHANGELOG. Co-authored-by: Paul Zgordan <dancingteeth@users.noreply.github.com>
When reviewSecondary errors after a successful primary review, log the fault, annotate review.md, and keep the primary verdict. Co-authored-by: Paul Zgordan <dancingteeth@users.noreply.github.com>
Classify worker faults, recycle the session, log failure context and partial Cursor usage, and advance to the next iteration. Fatal auth/config errors still abort. Park with agent_transient_exhausted when the consecutive transient cap is reached. Co-authored-by: Paul Zgordan <dancingteeth@users.noreply.github.com>
|
Reviewey Ray sensor (automated review, not a merge decision; the human owns merge) PR: #45 (draft) · push RiskHIGH What could go wrong?
Review depth
Pass 1b overlay (REVIEWS.md): "### HIGH — network egress, agentic tools, cline sdk, secondary judge, review gate". This PR changes secondary judge and review gate behavior, hence HIGH. Also: "Verifier is the hard gate; review is residual." The judge-down park keeps verify as the gate, which fits. §2b / §2c Lite: opened VerdictADVISORY Advisory
Nits
|
|
Reviewey Ray sensor (full Unified Code Review, triggered by HIGH risk; repo commit count 35). This is a sensor, not a merge decision: Paul owns merge. Agent loops may act only on RiskHIGH What could go wrong?
Review depth
Pass 1b (REVIEWS.md overlay): I'm quoting the laws that apply here. "Verifier is the hard gate; review is residual." "Gating impacts: VerdictADVISORY Advisory
Nits
|
|
Owner decision — per Paul in chat (2026-10-21 16:32 Tbilisi):
The other advisories from the sensor still stand: #45 (comment) — Reviewey Ray (sensor; human owns merge) |
Co-authored-by: Paul Zgordan <dancingteeth@users.noreply.github.com>
|
Reviewey Ray sensor (full Unified Code Review, triggered by HIGH risk; repo commit count 36). This is a sensor, not a merge decision: Paul owns merge. Agent loops may act only on RiskHIGH What could go wrong?
Review depth
Pass 1b (REVIEWS.md overlay): These laws apply: "Verifier is the hard gate; review is residual." "Gating impacts: Prior sensor items (#45 (comment)):
VerdictPASS Nits
|
Co-authored-by: Paul Zgordan <dancingteeth@users.noreply.github.com>
|
Reviewey Ray sensor (automated code-review sensor, not a merge decision; human owns merge). Push RiskHIGH What could go wrong?
Review depth
Overlay (REVIEWS.md, Pass 1b): "Shared helper / multi-caller change → open callee once; assumption mismatch → gating blocker with Agent-authored (Pass 2): co-authored commit. The intent comes from the prior sensor's nits (partial judge usage, VerdictPASS Nits
|
Co-authored-by: Paul Zgordan <dancingteeth@users.noreply.github.com>
|
Reviewey Ray sensor (short sensor, risk LOW; repo commit count 38). This is a sensor, not a merge decision: Paul owns merge. Agent loops may act only on RiskLOW What could go wrong?
Review depth
VerdictPASS Nits
|
…lain-text tool-call warning (#46) * loops: freeze lessons-file, prompt-rules, stall-probes bundles (Harness² ideas) * loops: judge follows worker runtime (opencode default judge model) * loops: run harness2-ideas loops on the Cursor runtime * feat(loop): prompt rules — Facts head, deliverable guard, plain-text tool-call warning (Harness² prompt-rules) - Stable ## Facts block states the verifier command and any deliverables. - Optional loop.json deliverables + deliverableNudges (default 2): the harness re-prompts the worker before verify while files are missing/empty. - Safety rule 7: never write a tool call as plain text; the next iteration gets a warning section when the previous reply contained one. * fix(loop): route worker faults in deliverable nudges through one fault path * fix(loop): confine deliverable paths to the repo Review (prompt-rules, should-fix): missingDeliverables joined repoRoot with the configured string, so an absolute path or a .. segment was stat'd as given and the nudge told the worker to create it outside the repo. - loop.json parse rejects absolute (POSIX/Windows) and escaping paths. - missingDeliverables never reports a path that is absolute or resolves outside repoRoot (defence in depth for callers that bypass the schema). - Tests for both. * test(doctor): scope the Telegram env reset to the one test that needs it The run-scope doctor test asserts the 'telegramNotify is enabled' warning, which only appears when Telegram env is unset. Under `doppler run` (how loops run verify.sh) AGENT_LOOP_TELEGRAM_* are set, so the test failed. The loop workers had worked around it by deleting the vars in a file-wide afterEach, leaving process env cleared for later tests (review should-fix, out of scope for the loops). Clear the three vars inside that test only and restore the caller's values in finally. * fix(cursor): treat Connect-node 'http/2 stream closed with error code CANCEL' as a detached HTTP/2 transport error stall-probes run 1 died with exit 1 in the JUDGE phase: @cursor/sdk 1.0.23's detached team-repo RPC (getTeamReposOrEmptyIfNotInTeam → checkRepoBlocked → shouldBlockRead) rejected with `[canceled] http/2 stream closed with error code CANCEL (0x8)`. The HTTP/2 unhandled-rejection guard exists for exactly this RPC but only matched NGHTTP2_* / ERR_HTTP2_STREAM_ERROR text, so it rethrew and Node exited before the judge-error classification (#45) could run. Match Connect-node's RST_STREAM wording in both the HTTP/2 guard pattern and the general transport pattern; tests use the observed error shape. * fix(loop): own-line Facts heading in the stable head; exact verify command Review should-fix: the later Facts join left a single newline before ## Workspace (git) when no Facts block was present, and glued the block onto safety rule 6 (renderLoopPromptRulesSection has no trailing newline). - Join the optional stable-head block (## Facts) after the rules with exactly one blank line; without it the head is byte-identical to main. - Tests assert every heading starts at line start after a blank line. - Nits: ## Facts named in the stable-head comment; the verifier command is shown exactly as configured (not trimmed). --------- Co-authored-by: dancingteeth <dancingteeth@users.noreply.github.com>
* loops: freeze lessons-file, prompt-rules, stall-probes bundles (Harness² ideas) * loops: judge follows worker runtime (opencode default judge model) * loops: run harness2-ideas loops on the Cursor runtime * feat(loop): prompt rules — Facts head, deliverable guard, plain-text tool-call warning (Harness² prompt-rules) - Stable ## Facts block states the verifier command and any deliverables. - Optional loop.json deliverables + deliverableNudges (default 2): the harness re-prompts the worker before verify while files are missing/empty. - Safety rule 7: never write a tool call as plain text; the next iteration gets a warning section when the previous reply contained one. * fix(loop): route worker faults in deliverable nudges through one fault path * fix(loop): confine deliverable paths to the repo Review (prompt-rules, should-fix): missingDeliverables joined repoRoot with the configured string, so an absolute path or a .. segment was stat'd as given and the nudge told the worker to create it outside the repo. - loop.json parse rejects absolute (POSIX/Windows) and escaping paths. - missingDeliverables never reports a path that is absolute or resolves outside repoRoot (defence in depth for callers that bypass the schema). - Tests for both. * test(doctor): scope the Telegram env reset to the one test that needs it The run-scope doctor test asserts the 'telegramNotify is enabled' warning, which only appears when Telegram env is unset. Under `doppler run` (how loops run verify.sh) AGENT_LOOP_TELEGRAM_* are set, so the test failed. The loop workers had worked around it by deleting the vars in a file-wide afterEach, leaving process env cleared for later tests (review should-fix, out of scope for the loops). Clear the three vars inside that test only and restore the caller's values in finally. * fix(cursor): treat Connect-node 'http/2 stream closed with error code CANCEL' as a detached HTTP/2 transport error stall-probes run 1 died with exit 1 in the JUDGE phase: @cursor/sdk 1.0.23's detached team-repo RPC (getTeamReposOrEmptyIfNotInTeam → checkRepoBlocked → shouldBlockRead) rejected with `[canceled] http/2 stream closed with error code CANCEL (0x8)`. The HTTP/2 unhandled-rejection guard exists for exactly this RPC but only matched NGHTTP2_* / ERR_HTTP2_STREAM_ERROR text, so it rethrew and Node exited before the judge-error classification (#45) could run. Match Connect-node's RST_STREAM wording in both the HTTP/2 guard pattern and the general transport pattern; tests use the observed error shape. * fix(loop): own-line Facts heading in the stable head; exact verify command Review should-fix: the later Facts join left a single newline before ## Workspace (git) when no Facts block was present, and glued the block onto safety rule 6 (renderLoopPromptRulesSection has no trailing newline). - Join the optional stable-head block (## Facts) after the rules with exactly one blank line; without it the head is byte-identical to main. - Tests assert every heading starts at line start after a blank line. - Nits: ## Facts named in the stable-head comment; the verifier command is shown exactly as configured (not trimmed). * feat(loop): lessons.md from green runs (Harness² lessons-file) After a green verifier run the harness writes deduped verify gotchas to <loop>/lessons.md (capped at 6000 chars); the next run injects them into the stable prompt head before ## Workspace (git). lessons.md is frozen (workers cannot edit it). Opt out with "lessons": false. Includes the lessons review fixes (own-line lessons heading in the stable head; trim to the cap using the configured verify command rather than re-parsing a backtick span). * feat(loop): stall probes — two guided workers on the first hard stall (Harness² stall-probes) Opt-in stallProbes (default false). On the first hard stall at escalateAfterStagnation the harness runs two sequential probe workers (procedure hints vs repo facts), snapshots/restores the git workspace between them, keeps the probe whose verify goes green or moves the failure fingerprint, logs probe: { variant, kept }, then allows normal model escalation on later stalls (at most once per run). * fix(loop): stall probes — git-object workspace snapshots and probe-round review fixes Stall-probes loop review (grok-4.7, BLOCKERS) found data-loss bugs in the default WorkspacePort (reset --hard + untracked delete wiped staged files under the loop bundle; porcelain-quoted paths reverted/deleted; symlinks followed on restore; staged renames half-restored; 0755 → 0644; UTF-8 re-encoding of binary bytes; HEAD tripwire after the writes). createGitWorkspacePort is rebuilt on git plumbing instead of a file copy: - snapshot is read-only: HEAD + `git write-tree` of the real index + a worktree tree built with `git add -A` in a throwaway GIT_INDEX_FILE (respects .gitignore, stores symlinks as links, keeps modes and raw bytes, -z plumbing so no path quoting). No stash, no ref, no file writes. - restore diffs the current worktree tree against the snapshot (diff-tree -r -z --no-renames), deletes then writes only the changed paths, never writes through a symlink (lstat/unlink the link itself, parents must be real dirs), restores 0755/0644 and symlinks, never deletes under .cursor/loops/, then `git read-tree` restores the index exactly (staged adds/renames/deletions). - HEAD check runs before any write: a probe commit on top of the snapshot is undone with reset --soft (reflog keeps it); an unrelated HEAD throws. agentLoop probe round (review should-fix / must-fix): - finalVerify runs after a green probe (from the loop's round-2 fix). - default stagnationThreshold no longer stops the run in the probe round itself, so escalation can resume (from the loop's round-2 fix). - a throwing procedure probe restores s0 before the shared fault path; a throwing facts probe restores the procedure attempt and logs it. - a snapshot failure (e.g. unmerged index) skips probes for the run instead of ending it. - budget stop after the probe round; probes get the deliverable nudges; a kept green probe is logged once (with its review), not twice. Guard tests: quoted/unicode paths, symlink to a file outside the repo, staged rename, 0755 file, staged new file under .cursor/loops, probe commit / unrelated HEAD, read-only snapshot, procedure probe that throws, single log line for a green probe. stallProbes stays off by default. * fix(loop): stall-probe restore keeps ignored files and the branch; park with both errors on a failed restore Review of #47 (BLOCKERS): - Blocker (data-loss): restore decided what to delete under the probe's ignore rules, so a probe that edited or deleted .gitignore made .env / node_modules look probe-added and they were deleted. Restore now puts the snapshot's ignore sources back first (every .gitignore in the change set, plus $GIT_DIR/info/exclude, captured in the snapshot), recomputes the diff under those rules, and never deletes a path that git check-ignore --no-index matches (also covers a probe that force-added an ignored file to the index). - HEAD tripwire also checks the branch: the snapshot records the symbolic ref; a probe's git checkout -b / --orphan / --detach is switched back (HEAD repointed, worktree untouched; a deleted branch is recreated at the snapshot commit) before the commit check runs. - A restore that throws no longer escapes the probe round or hides the worker error: it is wrapped in an AggregateError (restore error + original probe/worker error, cause = original) and the run parks (status waiting, failure domain stall_probe_restore_failed). - Nits: a restored regular file keeps its own permission bits (0600 stays 0600; only the exec bit follows git); a file/symlink parent under a protected dir is never unlinked. - Guard tests from the repro (edit/delete .gitignore, force-add, info/exclude, nested .gitignore, checkout -b with and without commits, orphan, detached, deleted branch, modes, protected parent; all fail on the previous code) and agentLoop tests for parking on a failed restore. * fix(loop): lessons.md opt-in, quoted as data, informative failure lines Review of #47: - lessons defaults to false (opt-in) — no new file in consumers' loop bundles on every green run in a patch release. - Verifier output is stored, and injected into the prompt, as a fenced block under a notice that it is data, not instructions. The fence is longer than any backtick run in the output so it cannot be closed early; older unfenced files are re-quoted when read into the prompt. - Gotchas are the most informative lines of each failure overcome instead of the first output line (always '[verify] 0 — toolchain' for this repo's verifiers): FAIL / error / assertion lines, stderr first, tagged with the failing [verify] step; banner, progress and summary lines are skipped; ANSI stripped; at most 3 lines of 240 chars per failure; the 6000-char file cap still drops the oldest. Frozen lessons-file acceptance updated for the default change: its wiring tests now opt in with lessons: true, and it asserts the default is false; the pinned sha256 in verify.sh is updated to match. No other verify.sh change. * docs: CHANGELOG/README for #47 review — snapshot limits, lessons opt-in, #46 entries - stallProbes: restore behaviour (branch/HEAD, ignore sources, parking on a failed restore) and its limits: ignored files are not snapshotted, and it is not safe with concurrent writers (one loop per worktree). - lessons: opt-in, quoted as data, informative lines. - Missing #46 entries: ## Facts block, plain-text tool-call warning + safety rule 7, HTTP/2 CANCEL crash fix. * loops(lessons-file): GOAL.md — lessons defaults to false (opt-in) Per Paul in chat 2026-10-21 19:15 (review of #47). verify.sh does not pin GOAL.md, so no sha change. --------- Co-authored-by: dancingteeth <dancingteeth@users.noreply.github.com>
- package.json 0.8.3 (lockfile has no root version; unchanged). - Version labels kept in sync: README current line, CONSUMERS.md, docs/embed-api.md, llms.txt, site/ pages + site/llms.txt, and the site version test. - CHANGELOG: Unreleased -> 0.8.3 — 2026-10-21 with PR links (#44 #45 #46 #47); adds the parked-runs-exit-2 and deliverables-confined entries. Co-authored-by: dancingteeth <dancingteeth@users.noreply.github.com>
Harness improvements for Cursor dogfood (composer worker, grok judge). Separate commits; no version bump; draft.
1. Verify: sidecar default (behaviour change)
verifyLogModedefaults tosidecar; tail-biased preview;verify-logs/gitignore guidance in README/init.2. Billing token breakdown on green runs
3. Judge resilience
reviewSecondaryfallback primary; parkreview_judge_failedwhen both fail.skipMergeSecondaryReviewso the same model does not run merge secondary.4. Tool-error stats
log.ndjsonandrun-report.md.5. Transient worker SDK errors
Exit codes (owner: Paul)
status: waitingexits2(same as main 0.8.2 incomplete), not0or3.run-batchunchanged (0|2).Verified
pnpm test,pnpm exec tsc -p tsconfig.json --noEmitOpen questions