Repository navigation
feat: multi-model PR review workflow - #24
Conversation
pr_review_multi.yml runs Opus, GPT, Kimi, GLM and DeepSeek as read-only agents on the PR, merges their findings, has a judge model check each one against the code, and posts one comment: a table of findings, collapsed duplicates, pre-existing and dismissed findings, housekeeping notes and the review's cost. Large PRs are split into parts and reviewed in priority order within a per-PR budget; '/review continue' finishes a partial review and '/review budget=N' raises the budget. The runner (ai-review/) is the pilot's orchestrator, extended for CI: keys from the environment, Claude through the Anthropic API, repository config read from the PR base branch, and the review cost in the comment. In this repository '/review' now runs the multi-model review; the single-model commands are unchanged.
Runner (ai-review/run_pilot.py): - --isolate shared: with a key other reviews use at the same time, the budget gate and watchdog count this run's own computed spend (Claude lanes included while they run) and skip key-usage settles. - --deadline-min: no new part of a split PR starts after the deadline; judging is skipped past it; a cancelled run ignores further signals and writes its report and comment, which is now written first. - --sandbox-check: before any paid call, one command in the Codex lane sandbox must run, read the checkout, fail to read a denied path and fail to find a canary in other processes' environments. The runner is undumpable (prctl), and AI_REVIEW_DENY adds CI paths to the deny lists; deny entries for missing paths are dropped (bubblewrap fails on them). - The Codex profile denies scratch entries one by one: denying the whole scratch dir broke git in every Codex lane. - continue: reuse only results of the same head and the same files per part, keep only well-formed findings, reuse housekeeping, never drop a fully reused part, and estimate only the lanes that run again. - PR comment: model text cannot inject HTML, mentions, links or images; the header says how many findings the judge checked and marks the rest unverified; a stopped run says why and exits non-zero. - The repository config's threat model must be a file next to it. - Finder prompt v2.7: bugs that hurt honest use count as much as attacks. Workflow (pr_review_multi.yml): continue only takes artifacts from this repository's own, non-fork runs of the calling workflow and extracts them safely; secrets, the fork check and the runner source are checked up front with the reason in the comment; GH_TOKEN only in the steps that need it; the runner is exec'd; a cancelled or failed review still posts something useful. ai-review.yml: call this repository's workflows by local reference; the 'latest' tag they used does not exist. README: correct budgets, config semantics and cost figures.
plan_units stored each chunk of an oversize area under a label built
from the basenames of its first and last file. Chunks with the same
pair of basenames ({mod.rs .. mod.rs} in a Rust module tree) got the
same key, so each one replaced the previous and only the last chunk
was reviewed. The other files were in no unit, and neither the comment
nor the coverage table reported them.
Label chunks with their paths below the area instead. Chunks are
disjoint runs of the sorted paths, so every chunk gets its own key.
…anges From a second multi-agent review of this PR: - A part whose reviewers failed (key out of credit, provider outage, timeout) is reported as partially or not reviewed, with the continue hint, instead of reviewed. - A Codex reviewer that times out or fails counts its estimated cost, not $0, so the budget gate, the watchdog and the comment see the spend. - The comment lists changed files no reviewer saw: binary to git, whitespace-only changes, unusual file names, and files placed in no part. - Python, shell, YAML and Makefiles keep whitespace changes in their counts and diffs. - seed.py: a tab or newline in a file name, a top-level `(root)/` directory, an unbalanced brace in a Rust test module, and sensitive docs no longer hide files or lines from the reviewers. - Model text in the comment: carriage returns and `&` are escaped. - The runner finds seed.py under `python -I`. - Offline tests (unit planning, coverage of changed files, the comment) run in the check workflow, which now also runs on changes to the review workflows. - Docs: what the sandbox check covers, the threat model's file type, the pinning fallback, running locally; this repository's priority list no longer matches every file.
The module docstring described the local pilot (Keychain only, cross-model verification per finding, docs lane). Help texts pointed at files only the pilot has (README item 8, finder_parallel.md) and left out `--claude-auth anthropic`, which CI uses. Comments said Claude Code lanes have no soft budget and that the judge is Opus. The dry run's spend table left out `--isolate shared`. The README now says ANTHROPIC_API_KEY is needed for `fable` too, and the `sandbox_check` input lists `off`.
Each changed file is now handled by the first rule that matches: always_review (reviewed as code), ignore, the built-in skips (lockfiles, binaries, generated and vendored code, whitespace-only changes, docs), tests, and everything else gets every reviewer. A part made only of low_risk code gets low_risk_lanes (default gpt, kimi, glm) in any PR, split or not, unless triage promotes it. Large PRs are reviewed in path order, low-risk parts last; `continue budget=N` reviews what the budget did not reach. Removed: sensitive, default_sensitive, priority (and its double budget), nonsensitive_lanes (now low_risk_lanes) and unit_depth. A config that still uses one of them is refused with what replaces it.
A label event runs the PR's own workflow files, so a PR that changes the review (this one) can review itself before merge. Fork PRs are refused, as for comments.
Most repositories don't need one. A missing threat model is an info line in the job log, and a warning only when the config names a file that is not there. A threat_model.md without a config.toml is now used too: the workflow passed no config, so the runner never looked for it.
AI security reviewThe review was cancelled or timed out before it wrote a report. Job log |
AI security review: 33 finding(s) to checkReviewed by Opus, GPT, Kimi, GLM, DeepSeek. A judge model checked 18 of 34 of the findings against the code; the others keep the reviewer's own label (unverified).
DetailsM2. Plaintext Git credentials remain readable by review agents (high, key/secret handling)
M1. Claude API key has no equivalent sandbox preflight check (high, key/secret handling)
M3. Git -w hides semantic changes inside string literals (medium, logic)
M5. Read-only users can authorize paid reviews and $100 budgets (medium, authz/authn)
M10. Continuation reuses stale reviews after the merge base changes (medium, logic)
M4. Renamed or copied files are classified only by their destination path, so moving code into an ignored/generated/test/docs path can bypass full review (medium, logic)
M7. Persistent shell sessions bypass the Codex soft-budget gate (medium, DoS/resource)
M9. Model finding floods can make deduplication quadratic (medium, DoS/resource)
M11. Budget gates discard successful reused lanes from partially reviewed units (medium, logic)
M14. Built-in generated/vendored globs silently drop real code from review and from the comment's 'Not reviewed' line (medium, supply chain)
M15. Rust #[cfg(test)] range check can be bypassed: production code after a fake test module inside a string literal is stripped from the diff (medium, logic)
M16. Rust test stripping also removes production items on the closing line (medium, logic)
M17. Test collection performs quadratic work on PR-controlled files (medium, DoS/resource)
M18. Deletion-only hunks of 15+ lines are replaced by a one-line summary, and under -W the whole enclosing function goes with them (medium, logic)
M20. Split PR test files can be placed in an unreviewed part and omitted from warnings (medium, logic)
M8. Claude Code reviewer lanes cannot run any Bash command on the Linux CI runner (bubblewrap fails to create a mount point inside the read-only checkout) (medium, logic)
M19. Generated chunk labels can overwrite real package areas (low, logic)
M21. Config check misses legacy keys rejected by the runner (low, logic)
M22. The workflow accepts any slash word as the command without checking it against inputs.command (low, logic)
M23. Spaced lane lists skip the Anthropic secret check (low, logic)
M26. An infinite line value in a finding raises OverflowError and discards all of that reviewer's findings for the part (low, logic)
M27. One housekeeping item with a non-integer line (e.g. "45-60") discards the whole housekeeping lane (low, logic)
M29. After a cancelled or aborted run, parts not yet finished keep status "planned" and are left out of the comment's partial-review list and
M30. Zero PR budget disables all review instead of disabling the budget (low, logic)
M31. Cancelled runs omit interrupted and unstarted parts from coverage warnings (low, logic)
M32. CODEOWNERS and dependency .txt files are classified as docs and kept out of the security review (low, logic)
M33. Glob matcher does not support standard character classes (low, logic)
M34. Renamed/copied deletion compaction is attributed to the old path, leaving compaction/effective-size totals wrong (low, logic)
M35. strip_test_modules drops a file's trailing newline when its last hunk ends inside a test module, gluing the next file's diff header onto the marker line (low, logic)
M36. Test-only PRs create units that can never be reviewed (low, logic)
M24. Re-running a failed review job may fail to upload its artifact, so the old review is posted (low, logic)
M25. Cost of harness lanes routed via a static preset is estimated from a dynamically chosen host that may not be in the preset (low, logic)
M28. Comment sanitizer's code/text parity tracking can be desynchronized with double-backtick spans, letting
Duplicates folded into another finding: 2
Pre-existing (real, but already on the base branch): 1
Housekeeping (not security): 2Stale docs, missing tests, leftovers and inconsistencies, from a separate cheap model. Not checked by the judge.
Cost of this review: ~$16.96 of a $50 budget (estimated from token counts). |
One review covers the whole PR. A PR over 20,000 effective lines is not reviewed: the comment gives its size and asks for it to be split. The splitting, part ordering, part map and per-part budget gate are gone; the default budget is one $30 per PR. On propamm #1 (one run per arm), one review of 10k or 21k lines found the critical and the same real highs as four ~2.5k-line parts at about half the cost. The prompt still holds 250k chars of diff: Codex compacts its context well below the models' 1M windows (GLM at a 408k-char prompt, GPT at 795k), so a bigger diff would be lost to its summary. Above 250k the diff is ranked, and a file too big to fit is cut where the room ends instead of being dropped. A Codex compaction is now logged and resets the in-flight cost estimate, which kept counting the dropped context. Reviewers' dollar limits grow with the prompt size. Malicious-code check: one cheap call (DeepSeek V4 Flash, GPT as fallback) over what the security reviewers do not see and what runs on a developer's machine or in CI by itself (tests, build and install scripts, hooks, editor tasks, dependency sources), plus the lines a pattern scan matched anywhere. Findings go at the top of the comment. On 10 planted attacks and 6 look-alike clean changes it caught 10 and flagged none; on 10 real merged PRs it raised one item. Housekeeping gets 150 commands (60 stopped it on a 5-line part).
In the first CI review (this PR, run 44151243197) four Codex lanes reached their soft limit and not one command was refused: DeepSeek ran 50 commands past its stop and was killed at the 35-minute timeout with no answer, twice. The gate's zsh files lived in the lane folder under the runs dir, which CI read-denies, with a more specific "read" entry for that folder. macOS honours the nested entry; Codex's Linux sandbox does not, so zsh could not read the gate. The gate files now live in ~/.ai-review-gates/<hash of the lane folder>, outside every denied path. The sandbox self-check also closes a gate and runs a zsh command through the lane sandbox: it logs whether the command was refused, so the next CI run shows the gate working on Linux before any paid call. Checked on macOS: the probe is refused.
It read only tests, files that run by themselves and files with a pattern hit, so a harmful helper in product code, called from an innocent-looking test line, was out of its view. It now reads every changed file but docs and lockfiles, files with a hit first, then tests and files that run by themselves, then the rest, up to 600k chars (one direct call: Codex's compaction does not apply; about a cent). Checked with variants written to avoid every pattern (paths and hosts built from pieces, hex-encoded commands, a `pretest` hook, Make variables, a product-code helper called from a test), planted into a real 5,200-line waas PR: 6 of 6 caught at high confidence, twice. The 16 corpus cases still score 10 of 10 caught, 0 of 6 flagged, and 10 real merged PRs get no item ($0.002-0.011 each).
…er compact the diff away Codex 0.142.2 knows none of the lanes' OpenRouter model strings. On its fallback metadata it caps the context window at 272k and compacts at 244,800 tokens whatever config.toml says, and compaction replaces the diff with a summary. Every Codex lane (finders, judge, housekeeping) now gets a one-entry model catalog (model_catalog_json) for the exact model string it sends, preset suffix included, with the model's OpenRouter context length as its window. Compaction is at 0.85 of it, and at most 0.97 of the hosts' max prompt where OpenRouter gives one (GPT-6.1-sol: 922k). The entry (ai-review/codex_catalog_entry.json) is a frozen copy of Codex 0.142.2's fallback metadata, base instructions included, so the requests stay the same; it must be re-captured when the pinned Codex version changes. In the pilot: 0 compactions, GPT up to 284k and GLM up to 345k tokens per request. Each lane also writes requests.jsonl (per-request token usage, Codex's effective window, compaction markers), and the runner logs a WARNING if a lane still compacts. With the compaction gone, the diff in the prompt goes from 250k to 600k chars (about 15,000 lines of whole functions, 150-250k tokens).
Its 262k-token window cannot hold a full 600k-char diff plus room to work; the default Kimi reviewer is K3.1 Pro (1M). Nobody selected it.
Compaction is left on as a backstop, but every one is now counted, so the prompt size can be tuned from real runs: per agentic lane (Codex and Claude Code, finders, judge and housekeeping) its compactions, its largest single request in tokens and its window, in summary.json, a "Context" section of the report and one line of the job log. Not in the PR comment.
…enced new names) fed into the prompt
…budget The script from the previous commit (missing paths and make targets, new names nothing references) is no longer meant for the housekeeping prompt. Measured on propamm #1 and actions#24 (one or two samples per variant): with the leads in its prompt the model only checked leads and stopped searching on its own, and an added "then search on your own" line did not change that. Without them, the old prompt found a real issue no lead covers. So the model keeps the old, short prompt, and the leads go to their own collapsed section of the comment, marked as not checked: on propamm #1 that is 17 leads, 16 real, including 2 that no model run found. Housekeeping's dollar limits now grow with the prompt size, as the finders' do: on a 711k-char prompt it stopped after ~38 of its 150 commands.
The cap was 20 findings, set when each finding had its own Claude session (~$0.39 each). The first CI review of actions#24 (36 findings) left 16 lows unjudged. The judge is now one GPT session (~$0.30), and 15 measured sessions of 3-14 findings used 9-22 commands and 2-5 minutes, about 1-2 commands per finding. So: cap 60, and the session's soft limit is 15 + 2 commands per finding (up to 150), 5 minutes + 20 s per finding (up to 25). One session also keeps duplicate detection across all findings.
AI security review: 16 finding(s) to checkReviewed by Opus, GPT, Kimi, GLM. A judge model checked every one of the findings against the code.
DetailsM1. Label-triggered reviews execute PR-controlled workflows with API secrets (high, key/secret handling)
M2. Reviewer sandbox omits common local credential stores (high, key/secret handling)
M3. Claude Bash path is not covered by the /proc deny (high, key/secret handling)
M15. Do not silently omit oversized files from the tool-less malicious check (medium, logic)
M10. Invalidate reused reviews when the merge-base changes (medium, logic)
M5. Codex artifacts can disclose unmasked local Git credentials (medium, key/secret handling)
M6. Hit truncation hides lockfile dependency sources from every reviewer (medium, logic)
M7. Preserve confirmed verdicts when folding judge duplicates (medium, logic)
M8. OpenRouter preflight counts Anthropic-billed Opus spend (medium, logic)
M9. Continue preflight does not subtract reusable reviewer cost (medium, logic)
M11. Whitespace filtering can hide semantic string changes (medium, logic)
M12.
M13. Raw brace counting can still hide production Rust as test code (medium, logic)
M14. Decode Git-quoted paths before scanning lockfile changes (medium, supply chain)
M16. A PR with no code files gets 'No reviewer finished' and is not reported as not reviewed (low, logic)
M17. Deletion-only hunks of 15+ lines are replaced, with their whole-function context, by a one-line note, hiding removed checks (low, logic)
Pre-existing (real, but already on the base branch): 1
Housekeeping (not security): 1Stale docs, missing tests, leftovers and inconsistencies, from a separate cheap model. Not checked by the judge.
Leads from a script (not checked): 4Found without a model, from the lines this PR adds. Some can be wrong, e.g. a path created at runtime or a name used through reflection.
Cost of this review: ~$8.49 of a $30 budget (estimated from token counts). |
…its answer file; deny registry and git credentials A lane whose last request ran out of output tokens while reasoning (DeepSeek, CI run 44227584384) ended with an empty answer and failed. It now gets one 'codex exec resume' in the same session that asks only for the JSON answer. last_message.txt is masked like stdout and stderr. ~/.npmrc, ~/.yarnrc.yml, ~/.git-credentials, ~/.pypirc and ~/.gem/credentials are denied to Codex and Claude lanes (one list). Removes the unused PROTECTED_CATEGORIES.
…ext module as test code
'#[cfg(test)] use super::*;' above a production 'pub mod x {' hid that module from the reviewers.
The attribute now counts only when it is alone on its line, or followed by attributes or a comment.
…hs are scanned The malicious-code check sees lockfiles only through their pattern hits. Those hits came after earlier files' hits and were lost past the 300-hit scan limit or the 150-hit prompt limit; they now come first and are always kept. The scan's diff no longer C-quotes non-ASCII paths, which it skipped.
Codex logs each model response's id with RUST_LOG=feedback_tags=info (OpenRouter's generation id; checked against a local stub: one line per response, no key or headers in them). After the lane, each id is looked up with OpenRouter's free generation endpoint; requests.jsonl gets one row per id with its provider and the lane notes say e.g. 'served by Baidu 20, Novita 8'. Covers finders, housekeeping and the judge, which all run through Codex.
Codex sends no output limit, so a provider's default applied: DeepSeek V4 Pro stopped at exactly 8,000 tokens while reasoning and gave no answer (CI run 44227584384). --setup-presets now stores max_tokens (32,000, or the model's maximum when lower) in each preset and warns if OpenRouter does not keep it; a lane whose preset has no limit says so in its notes. The presets must be re-created to take effect.
…w kept A finding the judge confirmed as a bug, marked duplicate_of one it dismissed, disappeared from the table. Each group of duplicates now keeps its strongest finding (Bug, then Suspected, Pre-existing, dismissed; then the higher severity) and lists the others as 'also'.
… same merge-base The pre-flight check held the whole estimate against OpenRouter credit, including Opus lanes billed by Anthropic (--claude-auth anthropic) and, on /review continue, lanes that would be reused. It now uses the OpenRouter-billed estimate of what will run (preflight.json keeps both). A continue reuses an earlier run only when the head and the merge-base both match: a retargeted PR has a different diff in the same files.
A file over the check's 600k-character input was left out silently and the comment still said 'nothing found'. The comment now says the check was partial and names those files.
… PR has no code 'Partial review: this PR was not fully reviewed (reviewers failed)' appeared when one reviewer failed and four reviewed the whole PR. The comment now says which reviewer did not finish and why in a few words (e.g. 'DeepSeek (empty answer) did not finish; Opus, GPT, Kimi, GLM reviewed the whole PR') with the continue hint. 'Partial review' is kept for code no reviewer finished on, and names that part. A PR with only tests, docs or other non-code files says it got no security review, instead of 'No reviewer finished'.
A fixed limit could cut off models that work today. The provider log shows which provider stops early, so that one can be dropped instead.
What
A new reusable workflow,
pr_review_multi.yml, runs a multi-model security review and posts one comment.curl | sh, credential paths, secret environment variables, dependencies from git URLs or tarballs). Anything it finds goes at the top of the comment./review budget=Nraises it, and/review continuefinishes a review that stopped at its budget or time limit.The runner is under
ai-review/. It is the orchestrator from our pilot (about 70 runs on ethrex, lambda_vm and a private repo), extended for CI:threat_model,ignore,test_globs,always_review(text such as prompts, reviewed as code) andlow_risk(code that gets three reviewers,low_risk_lanes, instead of five);In this repository,
/reviewnow runs the multi-model review, and so does adding a label starting withai-review. A label runs the PR's own workflow files, so this PR can review itself before merge. Every job inai-review.ymlnow calls this repository's workflows by local reference: thelatesttag they used does not exist, so every comment-triggered run has failed since 2026-11-04.What needs to be done
1. Before the first run (org admins)
OPENROUTER_API_KEY. Confirm that its OpenRouter account holds theair-*provider presets (Settings → Presets; seeai-review/presets.json), and that the key has a credit limit: it is the hard ceiling for every model except Opus. Reviewers call their models through those presets (model@preset/air-...), and presets belong to an account, so a key from another account fails on every call. If a model's endpoint list can't be read when a review starts, that reviewer runs unpinned for the whole review; the report in the artifact marks itUNPINNED.max_budget, $100, withbudget=N.ANTHROPIC_API_KEYsecret in both repositories belongs to a workspace with a monthly spend limit. Opus is billed there, outside the OpenRouter limit.2. First run
ai-reviewlabel to ci: switch /ai-review to the shared multi-model review lambda_vm#1032. Label events run that PR's own workflow, which calls this branch's commit. Comment triggers (/review,/ai-review) only work after merge, because they run the default branch's workflow.sandbox check: lane commands run, read the checkout, and cannot read other processes' environments. This is the first time the reviewers' sandboxes run on Linux. If the check fails, the review stops before any paid call, and the comment says why. Then either fix it, or accept the risk withsandbox_check: warnin the caller.8db585b; confirm on the next run, where the sandbox check now also logsthe soft-budget gate refuses commands inside the sandbox. In the log, Codex reviewers should either stop at their limit (soft budget stop ... commands refused from now on) or finish under it. If a reviewer runs to its hard cap instead, the zsh gate is not firing on Linux.run/units/*/agentic/*/events.jsonl), git commands should work in the reviewers' sandbox, with no permission errors.3. Merge and release
v2.1.0). This repository's/reviewuses the default branch, so it works right after the merge.uses:from this branch's commit (b265aff) to the release tag's commit and convert its.github/ai-review/config.tomlto the new keys (the runner now refusessensitive,default_sensitive,priority,nonsensitive_lanesandunit_depth), then merge it. That also puts lambda_vm's.github/ai-review/config onmain. Until then, reviews there use the defaults: every file gets the full review, no threat model.4. Follow-ups (not blocking)
max_budget($100) per request. Either check the commenter's write permission or lowermax_budget..github/ai-review/config.tomlandthreat_model.mdare optional; most repositories need neither. lambdaclass repositories use their own key and budget.Why
From the pilot (finder prompt v2.5, the five default models, judge GPT 6.1 Sol):
This PR went through a multi-agent review before opening:
The second commit is the result: 34 findings, and the 12 problems the judge found in the first round of fixes.
More commits came after opening:
fix: keep every file in the review when a split area is chunked #25, Julian's fix: when a split area was cut into chunks, chunks with the same label replaced each other and their files were never reviewed.
b265aff, from a second multi-agent review of the whole PR (six finders, a prosecutor and a defense for each finding, and a judge):(root)/directory or a Rust test module can no longer hide code from the reviewers;&in model text are escaped.That review found no way for a reviewer to read the API keys.
e60b8b1, a simpler repository config. Each changed file is handled by the first rule that matches:always_review,ignore, the built-in skips (lockfiles, binaries, generated and vendored code, whitespace-only changes, docs), tests, and everything else gets every reviewer. Removed:sensitive,default_sensitive,prioritywith its double per-reviewer budget,nonsensitive_lanes(nowlow_risk_lanes, default GPT, Kimi and GLM) andunit_depth; a config that still uses them is refused with what replaces them.6d71a49, one review per PR and the malicious-code check:8db585b, the soft-budget gate on Linux. In this PR's first CI review, four Codex reviewers reached a soft limit and none had a command refused: the gate's files sat under the runs directory, which CI hides from the sandbox, and Codex's Linux sandbox ignores the nested exception that macOS honours. DeepSeek ran on to the 35-minute timeout twice. The files now live outside every hidden path, and the sandbox check tests the gate before any paid call.Security and cost controls
sandbox_check: strictstops the review if it can.--isolate shared). Each review's budget gate and watchdog therefore count its own computed spend, Opus included while it runs, not the key's total.continue. It only reuses artifacts from this repository's own, non-fork runs of the same workflow and commit. They are extracted without symlinks.Testing
--isolate shared, repository config from a file):continuerun on the same commit: it reused the reviewer and the housekeeping and paid only for the judge ($0.16).ai-review-check.yml.6d71a49): the Linux sandbox check passed, all five reviewers and the judge ran, the comment was posted, ~$17. It found the soft-budget gate bug fixed in8db585b.Known limitations
continuere-runs the judge on every finding, about $0.30./procand the CI paths.