Skip to content

feat: multi-model PR review workflow - #24

Merged
MauroToscano merged 29 commits into
mainfrom
feat/multi-model-review
Nov 18, 2026
Merged

MauroToscano merged 29 commits into
mainfrom
feat/multi-model-review

Conversation

@MauroToscano

@MauroToscano MauroToscano commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

What

A new reusable workflow, pr_review_multi.yml, runs a multi-model security review and posts one comment.

  1. Reviewers. Opus 5.5, GPT 6.1 Sol, Kimi K3.1, GLM 5.3 and DeepSeek V4 Pro review the PR in parallel. Each is an agent in a read-only sandbox with no network, so it can read code outside the diff.
  2. Housekeeping. A cheap model lists stale docs, missing tests and leftovers.
  3. Malicious-code check. A cheap model (DeepSeek V4 Flash) reads what runs on a developer's machine or in CI by itself, which the security reviewers don't see: tests, build and install scripts, git hooks, editor tasks and dependency sources, plus every line a pattern scan matched (hidden Unicode, curl | sh, credential paths, secret environment variables, dependencies from git URLs or tarballs). Anything it finds goes at the top of the comment.
  4. Judge. Duplicates are merged, and GPT 6.1 Sol checks the findings against the code.
  5. Comment. One comment with:
    • a table of findings to check;
    • collapsed sections: duplicates, pre-existing bugs, dismissed findings with the judge's reasons, and housekeeping notes;
    • the changed files that no reviewer saw, if any;
    • the cost of the review.
  6. Size and budget. One review covers the whole PR, up to 20,000 effective changed lines. A bigger PR is not reviewed: the comment gives its size and asks for it to be split. The budget is $30 per review; /review budget=N raises it, and /review continue finishes a review that stopped at its budget or time limit.

The runner is under ai-review/. It is the orchestrator from our pilot (about 70 runs on ethrex, lambda_vm and a private repo), extended for CI:

  • keys come from the environment;
  • Opus is called through the Anthropic API;
  • the repository config is read from the PR's base branch. Every key is optional, and so is the threat model: threat_model, ignore, test_globs, always_review (text such as prompts, reviewed as code) and low_risk (code that gets three reviewers, low_risk_lanes, instead of five);
  • the review's cost is printed in the comment.

In this repository, /review now runs the multi-model review, and so does adding a label starting with ai-review. A label runs the PR's own workflow files, so this PR can review itself before merge. Every job in ai-review.yml now calls this repository's workflows by local reference: the latest tag they used does not exist, so every comment-triggered run has failed since 2026-11-04.

What needs to be done

1. Before the first run (org admins)

  • OpenRouter key. Both callers use the org secret OPENROUTER_API_KEY. Confirm that its OpenRouter account holds the air-* provider presets (Settings → Presets; see ai-review/presets.json), and that the key has a credit limit: it is the hard ceiling for every model except Opus. Reviewers call their models through those presets (model@preset/air-...), and presets belong to an account, so a key from another account fails on every call. If a model's endpoint list can't be read when a review starts, that reviewer runs unpinned for the whole review; the report in the artifact marks it UNPINNED.
  • Top up OpenRouter. A normal review costs about $3.40, one near the size limit about $7–12 (this PR's first review: ~$17, as 4 parts under the old code), at most $30, or max_budget, $100, with budget=N.
  • Anthropic. Make sure the ANTHROPIC_API_KEY secret in both repositories belongs to a workspace with a monthly spend limit. Opus is billed there, outside the OpenRouter limit.
  • OpenRouter privacy. Confirm the account's privacy setting still blocks providers that train on prompts. It applies to every key in the account.

2. First run

  • Run it. Done on this PR (run 44151243197, ~$17, 33 findings). Add the ai-review label to ci: switch /ai-review to the shared multi-model review lambda_vm#1032. Label events run that PR's own workflow, which calls this branch's commit. Comment triggers (/review, /ai-review) only work after merge, because they run the default branch's workflow.
  • Sandbox check. Passed on Linux. In the job log, look for: sandbox check: lane commands run, read the checkout, and cannot read other processes' environments. This is the first time the reviewers' sandboxes run on Linux. If the check fails, the review stops before any paid call, and the comment says why. Then either fix it, or accept the risk with sandbox_check: warn in the caller.
  • Soft budget. Failed on the first run, fixed in 8db585b; confirm on the next run, where the sandbox check now also logs the soft-budget gate refuses commands inside the sandbox. In the log, Codex reviewers should either stop at their limit (soft budget stop ... commands refused from now on) or finish under it. If a reviewer runs to its hard cap instead, the zsh gate is not firing on Linux.
  • Artifact. In the run's artifact (run/units/*/agentic/*/events.jsonl), git commands should work in the reviewers' sandbox, with no permission errors.
  • Comment. Checked on this PR: it shows the table of findings, the judge's count, and the cost line.

3. Merge and release

  • Merge this PR and tag a release (e.g. v2.1.0). This repository's /review uses the default branch, so it works right after the merge.
  • In ci: switch /ai-review to the shared multi-model review lambda_vm#1032, re-pin uses: from this branch's commit (b265aff) to the release tag's commit and convert its .github/ai-review/config.toml to the new keys (the runner now refuses sensitive, default_sensitive, priority, nonsensitive_lanes and unit_depth), then merge it. That also puts lambda_vm's .github/ai-review/ config on main. Until then, reviews there use the defaults: every file gets the full review, no threat model.

4. Follow-ups (not blocking)

  • Who can trigger and spend. Any org member or collaborator can trigger a review, even without write access, up to max_budget ($100) per request. Either check the commenter's write permission or lower max_budget.
  • Exact cost per review. Needs the cost-tracking proxy. Today a shared key's bill can't be split per review, so the comment shows an estimate from token counts.
  • More repositories. Each needs a caller workflow. .github/ai-review/config.toml and threat_model.md are optional; most repositories need neither. lambdaclass repositories use their own key and budget.

Why

From the pilot (finder prompt v2.5, the five default models, judge GPT 6.1 Sol):

  • Several models find more than one. Five PRs had a known, later-fixed bug:
    • lambda_vm #823: found by 5 of 5 models;
    • ethrex #6598: 4 of 5;
    • ethrex #5708: 4 of 5;
    • lambda_vm #353: 3 of 5;
    • ethrex #6680 (a stack overflow): found by none.
  • The judge filters noise without losing bugs. On 7 PRs with labelled findings, it dismissed all 15 planted false findings and no real bug. It agreed with independent prosecutor/defender/judge labels on 19 of 25 findings.
  • Cost. A PR reviewed in one part costs about $3, of which Opus is about 40%.

This PR went through a multi-agent review before opening:

  • four finders (workflow security, workflow correctness, the runner's CI changes, docs accuracy);
  • a prosecutor and a defender per finding;
  • a judge, who also checked the fixes.

The second commit is the result: 34 findings, and the 12 problems the judge found in the first round of fixes.

More commits came after opening:

  • fix: keep every file in the review when a split area is chunked #25, Julian's fix: when a split area was cut into chunks, chunks with the same label replaced each other and their files were never reviewed.

  • b265aff, from a second multi-agent review of the whole PR (six finders, a prosecutor and a defense for each finding, and a judge):

    • a part whose reviewers failed (key out of credit, outage, timeout) is reported as partially or not reviewed, instead of reviewed;
    • a Codex reviewer that times out or fails counts its estimated cost, not $0;
    • the comment lists changed files that no reviewer saw;
    • whitespace changes in Python, shell, YAML and Makefiles are reviewed;
    • odd file names, a (root)/ directory or a Rust test module can no longer hide code from the reviewers;
    • carriage returns and & in model text are escaped.

    That review found no way for a reviewer to read the API keys.

  • e60b8b1, a simpler repository config. Each changed file is handled by the first rule that matches: always_review, ignore, the built-in skips (lockfiles, binaries, generated and vendored code, whitespace-only changes, docs), tests, and everything else gets every reviewer. Removed: sensitive, default_sensitive, priority with its double per-reviewer budget, nonsensitive_lanes (now low_risk_lanes, default GPT, Kimi and GLM) and unit_depth; a config that still uses them is refused with what replaces them.

  • 6d71a49, one review per PR and the malicious-code check:

    • No more splitting. On propamm feat: use kimi k2.5 #1 (one run per arm), one review of 10,000 or 21,000 lines found the critical and the same real highs as four ~2,500-line parts, at about half the cost; the parts found only extra low-severity issues, all low or not real once checked. Outside lambda_vm, almost no PR in the last year was over 10,000 effective lines. Splitting, part ordering, the part map and the per-part budget gate are gone. Over 20,000 lines the comment asks for a split.
    • The prompt still holds 250,000 characters of diff. Codex compacts its context well below the models' 1M windows (GLM at a 408k-character prompt, GPT at 795k), replacing the diff with a summary; Opus did not. Above the limit the diff is ranked, and a file too big to fit is cut instead of dropped. A compaction is now logged.
    • Malicious-code check. On 10 planted attacks and 6 look-alike clean changes (built from a copy of waas) it caught all 10 and flagged none of the 6; three of the attacks, buried in a real 5,200-line PR, were all caught. On 10 real merged PRs it raised one item, a dependency from a personal fork. Under a cent per PR.
    • Reviewers' dollar limits grow with the prompt size; housekeeping gets 150 commands.
  • 8db585b, the soft-budget gate on Linux. In this PR's first CI review, four Codex reviewers reached a soft limit and none had a command refused: the gate's files sat under the runs directory, which CI hides from the sandbox, and Codex's Linux sandbox ignores the nested exception that macOS honours. DeepSeek ran on to the 35-minute timeout twice. The files now live outside every hidden path, and the sandbox check tests the gate before any paid call.

Security and cost controls

  • Token separation. The job that runs the models has read permissions only. Only the job that posts the comment can write, and it never runs a model or sees a key. Fork PRs are blocked before any secret is used.
  • Sandbox check. Before any paid call, a command runs in the reviewers' sandbox (no model) and must not be able to read other processes' environments. sandbox_check: strict stops the review if it can.
  • Lane environment and deny lists. Reviewers get a minimal environment. They cannot read the runner checkout, the runner temp dir (earlier review, config) or the Actions runner directory. The runner process is undumpable.
  • Model text in the comment. HTML, mentions, images and Markdown links are neutralized. Bare URLs stay, and GitHub links them.
  • Budget accounting. The key is shared by several repositories (--isolate shared). Each review's budget gate and watchdog therefore count its own computed spend, Opus included while it runs, not the key's total.
  • Hard ceilings. The OpenRouter key's credit limit covers every model except Opus. The Anthropic workspace's spend limit covers Opus.
  • Time. The judge is skipped if the reviewers end more than 285 minutes after the start, so the comment is posted inside the job's 330-minute limit. A cancelled run still writes its report and comment.
  • continue. It only reuses artifacts from this repository's own, non-fork runs of the same workflow and commit. They are extracted without symlinks.

Testing

  • Local runs of the runner in CI mode (--isolate shared, repository config from a file):
    • a dry run;
    • a paid smoke run on lambda_vm #823 with one reviewer: it found the known bug, and the judge confirmed it, for $0.69;
    • a continue run on the same commit: it reused the reviewer and the housekeeping and paid only for the judge ($0.16).
  • On macOS, with its Linux-only switch bypassed by hand, the sandbox check passed, and it failed as it should when the profile did not deny the runs directory. In the workflow it runs only on Linux, where it has not run yet: see "First run" above.
  • 27 offline tests (planning, coverage of changed files, the config rules, the size limit, the malicious-code scan, the comment) run in ai-review-check.yml.
  • The first CI review ran on this PR (with the code before 6d71a49): the Linux sandbox check passed, all five reviewers and the judge ran, the comment was posted, ~$17. It found the soft-budget gate bug fixed in 8db585b.

Known limitations

  • In CI, the cost in the comment is computed from token counts. The shared key's bill can't be split per review.
  • A continue re-runs the judge on every finding, about $0.30.
  • The sandbox check covers the Codex sandbox only. For Claude Code we rely on its own sandbox: from its 2.1.323 code, commands run under bubblewrap with a separate PID namespace (not yet confirmed on Linux). Its file tools are denied /proc and the CI paths.

pr_review_multi.yml runs Opus, GPT, Kimi, GLM and DeepSeek as read-only
agents on the PR, merges their findings, has a judge model check each one
against the code, and posts one comment: a table of findings, collapsed
duplicates, pre-existing and dismissed findings, housekeeping notes and
the review's cost. Large PRs are split into parts and reviewed in priority
order within a per-PR budget; '/review continue' finishes a partial review
and '/review budget=N' raises the budget.

The runner (ai-review/) is the pilot's orchestrator, extended for CI: keys
from the environment, Claude through the Anthropic API, repository config
read from the PR base branch, and the review cost in the comment.

In this repository '/review' now runs the multi-model review; the
single-model commands are unchanged.
Runner (ai-review/run_pilot.py):
- --isolate shared: with a key other reviews use at the same time, the
  budget gate and watchdog count this run's own computed spend (Claude
  lanes included while they run) and skip key-usage settles.
- --deadline-min: no new part of a split PR starts after the deadline;
  judging is skipped past it; a cancelled run ignores further signals
  and writes its report and comment, which is now written first.
- --sandbox-check: before any paid call, one command in the Codex lane
  sandbox must run, read the checkout, fail to read a denied path and
  fail to find a canary in other processes' environments. The runner is
  undumpable (prctl), and AI_REVIEW_DENY adds CI paths to the deny lists;
  deny entries for missing paths are dropped (bubblewrap fails on them).
- The Codex profile denies scratch entries one by one: denying the whole
  scratch dir broke git in every Codex lane.
- continue: reuse only results of the same head and the same files per
  part, keep only well-formed findings, reuse housekeeping, never drop a
  fully reused part, and estimate only the lanes that run again.
- PR comment: model text cannot inject HTML, mentions, links or images;
  the header says how many findings the judge checked and marks the rest
  unverified; a stopped run says why and exits non-zero.
- The repository config's threat model must be a file next to it.
- Finder prompt v2.7: bugs that hurt honest use count as much as attacks.

Workflow (pr_review_multi.yml): continue only takes artifacts from this
repository's own, non-fork runs of the calling workflow and extracts them
safely; secrets, the fork check and the runner source are checked up
front with the reason in the comment; GH_TOKEN only in the steps that
need it; the runner is exec'd; a cancelled or failed review still posts
something useful.

ai-review.yml: call this repository's workflows by local reference; the
'latest' tag they used does not exist.

README: correct budgets, config semantics and cost figures.
plan_units stored each chunk of an oversize area under a label built
from the basenames of its first and last file. Chunks with the same
pair of basenames ({mod.rs .. mod.rs} in a Rust module tree) got the
same key, so each one replaced the previous and only the last chunk
was reviewed. The other files were in no unit, and neither the comment
nor the coverage table reported them.

Label chunks with their paths below the area instead. Chunks are
disjoint runs of the sorted paths, so every chunk gets its own key.
…anges

From a second multi-agent review of this PR:
- A part whose reviewers failed (key out of credit, provider outage, timeout) is reported as
  partially or not reviewed, with the continue hint, instead of reviewed.
- A Codex reviewer that times out or fails counts its estimated cost, not $0, so the budget gate,
  the watchdog and the comment see the spend.
- The comment lists changed files no reviewer saw: binary to git, whitespace-only changes,
  unusual file names, and files placed in no part.
- Python, shell, YAML and Makefiles keep whitespace changes in their counts and diffs.
- seed.py: a tab or newline in a file name, a top-level `(root)/` directory, an unbalanced brace
  in a Rust test module, and sensitive docs no longer hide files or lines from the reviewers.
- Model text in the comment: carriage returns and `&` are escaped.
- The runner finds seed.py under `python -I`.
- Offline tests (unit planning, coverage of changed files, the comment) run in the check
  workflow, which now also runs on changes to the review workflows.
- Docs: what the sandbox check covers, the threat model's file type, the pinning fallback,
  running locally; this repository's priority list no longer matches every file.
The module docstring described the local pilot (Keychain only, cross-model verification per
finding, docs lane). Help texts pointed at files only the pilot has (README item 8,
finder_parallel.md) and left out `--claude-auth anthropic`, which CI uses. Comments said Claude
Code lanes have no soft budget and that the judge is Opus. The dry run's spend table left out
`--isolate shared`. The README now says ANTHROPIC_API_KEY is needed for `fable` too, and the
`sandbox_check` input lists `off`.
Each changed file is now handled by the first rule that matches: always_review
(reviewed as code), ignore, the built-in skips (lockfiles, binaries, generated
and vendored code, whitespace-only changes, docs), tests, and everything else
gets every reviewer. A part made only of low_risk code gets low_risk_lanes
(default gpt, kimi, glm) in any PR, split or not, unless triage promotes it.
Large PRs are reviewed in path order, low-risk parts last; `continue
budget=N` reviews what the budget did not reach.

Removed: sensitive, default_sensitive, priority (and its double budget),
nonsensitive_lanes (now low_risk_lanes) and unit_depth. A config that still
uses one of them is refused with what replaces it.
A label event runs the PR's own workflow files, so a PR that changes the
review (this one) can review itself before merge. Fork PRs are refused, as
for comments.
Most repositories don't need one. A missing threat model is an info line in
the job log, and a warning only when the config names a file that is not
there. A threat_model.md without a config.toml is now used too: the workflow
passed no config, so the runner never looked for it.
@MauroToscano MauroToscano added the ai-review Run the multi-model AI review on this PR label Nov 17, 2026
@github-actions

Copy link
Copy Markdown

AI security review

The review was cancelled or timed out before it wrote a report. Job log

@MauroToscano MauroToscano added ai-review Run the multi-model AI review on this PR and removed ai-review Run the multi-model AI review on this PR labels Nov 17, 2026
@github-actions

Copy link
Copy Markdown

AI security review: 33 finding(s) to check

Reviewed by Opus, GPT, Kimi, GLM, DeepSeek. A judge model checked 18 of 34 of the findings against the code; the others keep the reviewer's own label (unverified).
Partial review: 2 of 4 parts of this PR were not fully reviewed (reviewers failed): .github/ai-review + .github/workflows, ai-review/{run_pilot.py}. Comment /review continue to review only the missing parts, with a new budget; finished parts are reused.

# Finding Severity Status Found by Where
M2 Plaintext Git credentials remain readable by review agents high Bug GPT, Kimi (2/5) ai-review/run_pilot.py:1312
M1 Claude API key has no equivalent sandbox preflight check high Suspected GLM (1/5) .github/workflows/pr_review_multi.yml:287
M3 Git -w hides semantic changes inside string literals (also M12, M13) medium Bug Opus, Kimi, GLM (3/5) ai-review/seed.py:110
M5 Read-only users can authorize paid reviews and $100 budgets medium Bug Opus, Kimi (2/5) .github/workflows/ai-review.yml:30
M10 Continuation reuses stale reviews after the merge base changes medium Bug GLM, Kimi (2/5) ai-review/run_pilot.py:4374
M4 Renamed or copied files are classified only by their destination path, so moving code into an ignored/generated/test/doc medium Bug DeepSeek (1/5) ai-review/seed.py:251
M7 Persistent shell sessions bypass the Codex soft-budget gate medium Bug GPT (1/5) ai-review/run_pilot.py:1300
M9 Model finding floods can make deduplication quadratic medium Bug GLM (1/5) ai-review/run_pilot.py:2023
M11 Budget gates discard successful reused lanes from partially reviewed units medium Bug Kimi (1/5) ai-review/run_pilot.py:4424
M14 Built-in generated/vendored globs silently drop real code from review and from the comment's 'Not reviewed' line medium Bug Opus (1/5) ai-review/seed.py:265
M15 Rust #[cfg(test)] range check can be bypassed: production code after a fake test module inside a string literal is strip medium Bug Opus (1/5) ai-review/seed.py:376
M16 Rust test stripping also removes production items on the closing line medium Bug GPT (1/5) ai-review/seed.py:376
M17 Test collection performs quadratic work on PR-controlled files medium Bug Kimi (1/5) ai-review/seed.py:463
M18 Deletion-only hunks of 15+ lines are replaced by a one-line summary, and under -W the whole enclosing function goes with medium Bug Opus (1/5) ai-review/seed.py:542
M20 Split PR test files can be placed in an unreviewed part and omitted from warnings medium Bug GLM (1/5) ai-review/seed.py:864
M8 Claude Code reviewer lanes cannot run any Bash command on the Linux CI runner (bubblewrap fails to create a mount point medium Suspected Opus (1/5) ai-review/run_pilot.py:1602
M19 Generated chunk labels can overwrite real package areas low Bug DeepSeek, GPT (2/5) ai-review/seed.py:814
M21 Config check misses legacy keys rejected by the runner low Bug (unverified) GLM (1/5) .github/workflows/ai-review-check.yml:35
M22 The workflow accepts any slash word as the command without checking it against inputs.command low Bug (unverified) Opus (1/5) .github/workflows/pr_review_multi.yml:124
M23 Spaced lane lists skip the Anthropic secret check low Bug (unverified) GLM (1/5) .github/workflows/pr_review_multi.yml:153
M26 An infinite line value in a finding raises OverflowError and discards all of that reviewer's findings for the part low Bug (unverified) Opus (1/5) ai-review/run_pilot.py:937
M27 One housekeeping item with a non-integer line (e.g. "45-60") discards the whole housekeeping lane low Bug (unverified) Opus (1/5) ai-review/run_pilot.py:1970
M29 After a cancelled or aborted run, parts not yet finished keep status "planned" and are left out of the comment's partial low Bug (unverified) Opus (1/5) ai-review/run_pilot.py:3332
M30 Zero PR budget disables all review instead of disabling the budget low Bug (unverified) GLM (1/5) ai-review/run_pilot.py:3973
M31 Cancelled runs omit interrupted and unstarted parts from coverage warnings low Bug (unverified) Kimi (1/5) ai-review/run_pilot.py:4731
M32 CODEOWNERS and dependency .txt files are classified as docs and kept out of the security review low Bug (unverified) Opus (1/5) ai-review/seed.py:42
M33 Glob matcher does not support standard character classes low Bug (unverified) GLM (1/5) ai-review/seed.py:88
M34 Renamed/copied deletion compaction is attributed to the old path, leaving compaction/effective-size totals wrong low Bug (unverified) DeepSeek (1/5) ai-review/seed.py:530
M35 strip_test_modules drops a file's trailing newline when its last hunk ends inside a test module, gluing the next file's low Bug (unverified) Opus (1/5) ai-review/seed.py:588
M36 Test-only PRs create units that can never be reviewed low Bug (unverified) Kimi (1/5) ai-review/seed.py:786
M24 Re-running a failed review job may fail to upload its artifact, so the old review is posted low Suspected (unverified) Opus (1/5) .github/workflows/pr_review_multi.yml:349
M25 Cost of harness lanes routed via a static preset is estimated from a dynamically chosen host that may not be in the pres low Suspected (unverified) Opus (1/5) ai-review/presets.json:66
M28 Comment sanitizer's code/text parity tracking can be desynchronized with double-backtick spans, letting @user enti low Suspected (unverified) Opus (1/5) ai-review/run_pilot.py:2795
Details

M2. Plaintext Git credentials remain readable by review agents (high, key/secret handling)

  • Trigger: A developer runs the supported local review command with Git's credential-store helper and credentials in its default ~/.git-credentials file. Attacker-controlled PR content induces a reviewer to read that file, for example with base64 ~/.git-credentials. This does not require credentials to exist on fresh GitHub-hosted runners.
  • Mechanism: permissions_toml() extends the read-only profile, which permits reads outside the explicit deny list. Neither this list, scratch_denies(), nor the default environment denies cover ~/.git-credentials. The separate sensitive-path list in claude_settings() omits it too. Minimal child environments do not protect credentials persisted on disk. Output masking happens after agent execution and cannot prevent tool results reaching the model; encoding the contents also bypasses the token-pattern masking.
  • Impact: A prompt-injected review agent can disclose the developer's stored Git passwords or access tokens to the model provider or encode them into its review output. Those credentials can grant access to repositories beyond the PR being reviewed.
  • Judge: Both reviewer profiles omit ~/.git-credentials while retaining the developer's home directory and allowing reads outside explicit denials. A user-readable credential-store file is therefore accessible. Environment scrubbing does not protect it, and post-execution masking cannot prevent tool output from reaching the model provider.

M1. Claude API key has no equivalent sandbox preflight check (high, key/secret handling)

  • Trigger: A PR author controls text that becomes part of the reviewer prompt and can instruct the Opus/Fable lane, through prompt injection, to run a Bash command that reads and encodes the Anthropic key.
  • Mechanism: The workflow injects ANTHROPIC_API_KEY into the review process, and run_pilot.py passes it to Claude Code in run_claude. The preflight sandbox_selfcheck launches only a Codex sandbox; there is no equivalent check that Claude Code's Bash tool cannot read the Claude process environment or retain the key in its child environment. If the pinned Claude Code sandbox does not isolate /proc or scrub the key, the model can read it and return an encoded form; exact-string masking would not catch an encoded value, and the generated PR comment permits bare URLs.
  • Impact: The Anthropic API key could be exposed in a public PR comment and used up to the workspace spend limit.
  • Judge: The Anthropic key reaches Claude Code, and the preflight checks only Codex. However, Claude enables its sandbox and subprocess-environment scrubbing. Whether sandboxed Bash can recover the key depends on the pinned CLI's implementation, which cannot be confirmed from this repository. Encoded output would evade the runner's masking if key access succeeds.

M3. Git -w hides semantic changes inside string literals (medium, logic)

  • Trigger: A PR author changes only whitespace inside a string literal in a non-exempt code file, such as changing "https://good.example" to "https://good.example " in Rust, Go, JavaScript, or another language not listed in WS_EXTS. The change enters run_pilot.build_plan through seed.classify_files and seed.seed_diff.
  • Mechanism: classify_files runs git diff --numstat -z -M -C -w; Git treats the two string literals as equal under --ignore-all-space, so added and deleted counts become zero and line 270 classifies the file as a whitespace-only change. For a file that also has other changes, _git_diff passes -w, so the string-literal hunk is omitted from the reviewer seed. I confirmed with git diff --no-index --numstat -w: the raw diff reports 1 1, while the -w diff reports nothing. No later check recovers the hidden hunk.
  • Impact: A one-character semantic change to a URL, command, configuration string, regex, or other data can be silently excluded from all reviewer prompts and reported as merely whitespace-only, defeating the review coverage guarantee and allowing intentional review evasion.
  • Judge: Git's whitespace suppression is syntax-unaware, so whitespace changes inside string literals can be classified as whitespace-only. Mixed changes also lose those edits from the seed diff. This is a confirmed review-coverage defect, rather than a direct runtime vulnerability.

M5. Read-only users can authorize paid reviews and $100 budgets (medium, authz/authn)

  • Trigger: An organization member without repository write access, or an outside collaborator with only read or triage access, comments /review budget=100 on a PR. A triage-only collaborator can also add an ai-review* label to start a default-budget review.
  • Mechanism: author_association establishes membership or collaboration, not write permission. The new multi-review job uses this check as its only commenter authorization, while pr_review_multi.yml accepts the requested budget after validating only that it is numeric and at most max_budget (100 by default). Neither the reusable workflow nor the label path performs a repository-permission check. Concurrency limits simultaneous reviews but does not limit repeated paid requests.
  • Impact: Users who cannot modify the repository can authorize up to $100 of OpenRouter and Anthropic spend per request and repeat requests to exhaust shared credits or create unapproved charges. Label requests likewise allow triage-only users to initiate the automatic $20/$50 review budgets.
  • Judge: The new multi-review path accepts association rather than repository write permission, and budget=100 passes without further authorization. Its new label path also permits triage-level triggering. The association policy already existed for single-model reviews, but this PR adds caller-selected multi-review budgets and a new triage-accessible spending path. Repeated requests remain possible, althou

M10. Continuation reuses stale reviews after the merge base changes (medium, logic)

  • Trigger: The base branch advances after a review, and a user runs /review continue at the unchanged PR head. The new merge-base diff still assigns the same file list to a unit.
  • Mechanism: Reuse checks only the old head SHA and each unit's file names. It never compares old_sum["base"] with the newly computed mb, although the reviewed diff is base...head. Advancing the base can change hunks within the same files, including introducing reversals of newly landed base changes, without changing either value that is checked. reused_row therefore accepts results produced for a different diff.
  • Impact: New or changed hunks can receive no fresh review while the unit is presented as reviewed, and stale findings can be carried into the new comment.
  • Judge: Reuse validates only the head SHA and unit file lists, not the merge-base or review configuration/prompt identity. Results can therefore be reused for a changed diff or changed review semantics. Unrelated base advancement alone need not change the merge-base, but actual merge-base changes and the other report's configuration/prompt changes remain reachable triggers.

M4. Renamed or copied files are classified only by their destination path, so moving code into an ignored/generated/test/docs path can bypass full review (medium, logic)

  • Trigger: A PR author renames or copies a source file into a path matched by generated/vendored, repo ignore, tests, or docs rules (for example src/auth.rs -> vendor/auth.rs, or an always_review prompt -> docs/guide.md) while changing its contents.
  • Mechanism: git diff --name-status -z -M -C returns rename/copy records keyed by the destination path with the old path stored in old, but classify_files applies hit_ignore, hit_gen, keep, docs, and test routing only to the destination path. Only the --paths filter consults old. The entire rename/copy record is therefore treated as a filtered/docs/test file, even though the substantive change includes removal/modification of code from the old reviewed location. In split PRs the filtered record has no unit, so it is absent from any unit's OMITTED/PR-map prompt and is not flagged by not_reviewed.
  • Impact: A malicious or large PR can hide a code change from the full multi-model reviewers by moving it into a directory the seed skips. The comment routes it as vendored/generated/test/docs rather than warning unconditionally that no full reviewer saw it, so the review workflow's coverage guarantee is bypassed.
  • Judge: Routing and always_review matching use only the destination, whereas the explicit paths filter also checks old_path. A modified source rename into vendor therefore bypasses full review, including an old-path always_review rule. Filtered records disappear from split-unit omission lists; docs/test destinations can still receive their limited treatment.

M7. Persistent shell sessions bypass the Codex soft-budget gate (medium, DoS/resource)

  • Trigger: A Codex reviewer, potentially following instructions embedded in the PR, opens an interactive shell through exec_command with a PTY before its budget is exhausted, then sends further read commands through write_stdin after the budget flag is created.
  • Mechanism: The refusal check executes only when a new zsh process reads .zshenv. Commands sent to an already-running shell do not execute that startup file, so they continue to run after the flag exists. CodexTail._check_soft() does not independently stop commands after reaching the soft limit: its escalation requires more than five outputs beginning with GATE_MARK. The persistent-session path produces no such refusals. The installed Codex tool interface supports both persistent exec sessions and write_stdin, and run_codex() does not disable them or gate session input.
  • Impact: The advertised command, dollar, and time soft limits are bypassable, allowing a reviewer to keep investigating and spending until a separate hard lane/run cap or timeout intervenes. The read-only filesystem sandbox remains intact, but it does not enforce these budget limits.
  • Judge: The refusal gate runs only during zsh startup. Input sent to a shell session opened before exhaustion does not rerun it. Soft-budget escalation requires gate-prefixed refusal outputs, so persistent-session commands can continue without triggering that escalation. Separate hard cost caps and timeouts still apply.

M9. Model finding floods can make deduplication quadratic (medium, DoS/resource)

  • Trigger: A PR author uses the diff or PR text to prompt-inject reviewers into emitting the maximum number of very small findings; a split PR multiplies the number of lanes and units that contribute findings.
  • Mechanism: The findings schema has no maxItems, normalize_findings does not cap the finding count, and this deduplication compares every pair and runs difflib.SequenceMatcher for pairs that are not at the same location. Although each lane has an output-token limit, many lanes and units can still produce thousands of findings. The CPU-heavy loop runs before the merge model call and has no local timeout; the spend watchdog cannot stop it.
  • Impact: The review job can spend minutes or hours in quadratic comparisons and be killed by the workflow timeout before writing a comment, wasting the already-paid reviewer calls.
  • Judge: Accepted finding counts are uncapped, and distinct findings survive exact deduplication into an unconditional all-pairs SequenceMatcher loop. Findings accumulate across lanes and units. Neither grouping loop checks cancellation or a timeout, and the watchdog only stops registered subprocesses, so an accepted finding flood causes quadratic CPU work. Exact delay depends on flood size; SequenceMatche

M11. Budget gates discard successful reused lanes from partially reviewed units (medium, logic)

  • Trigger: A continuation has a unit with some successful reusable reviewers and some reviewers still to run, but the remaining work is blocked by the new budget, deadline, run cap, or an earlier unit exhausting the budget.
  • Mechanism: fresh is nonempty whenever even one lane is missing. Every gate then executes continue before the loop at lines 4435-4448 that appends reused rows to lane_rows. Consequently none of that unit's successful old results reach all_findings, the merged report, or the new artifact. The deadline and run-cap branches have the same ordering.
  • Impact: Previously completed findings disappear from the continuation's comment, and a unit with valid prior coverage is relabeled simply as not reviewed instead of retaining its partial results.
  • Judge: For a partially reusable unit, fresh is nonempty. Every listed budget, deadline, or run-cap gate can continue before reusable rows are copied into the new run. Those successful prior findings consequently disappear from the continuation's merged findings and artifacts, while the unit is labeled not reviewed.

M14. Built-in generated/vendored globs silently drop real code from review and from the comment's 'Not reviewed' line (medium, supply chain)

  • Trigger: A PR changes code under **/vendor/** (Go vendored modules, Rust [patch] path crates), **/third_party/**, **/dist/** or **/generated/**. Or it adds a handwritten file that happens to match **/*_generated.*, e.g. src/is_generated.rs, since * matches 'is'. Or a malicious author puts code in src/generated/x.rs and adds mod generated;.
  • Mechanism: classify_files marks every GENERATED_GLOBS hit as kind 'filtered' unless the repository lists it in always_review. Filtered files are not in the seed diff. not_reviewed() only reports odd names, binaries, whitespace-only changes and unplaced code/test files, so pr_comment's 'Not reviewed' line leaves them out. They appear only in the artifact report (write_report) and in the finder prompt's OMITTED list.
  • Impact: Tampering with vendored dependencies (a classic supply-chain route) or real code in a matching path gets no review. The PR comment, which promises to list changed files no reviewer saw, does not mention them, so humans may assume full coverage.
  • Judge: The generated/vendor exclusion is documented and intentional, but matching is purely path-based, so executable vendored code and handwritten matching files are excluded too. The reporting claim also holds: these exclusions are absent from Not reviewed. Their omission list is available for unsplit reviews, but split-unit prompts exclude unassigned filtered records.

M15. Rust #[cfg(test)] range check can be bypassed: production code after a fake test module inside a string literal is stripped from the diff (medium, logic)

  • Trigger: A PR author adds Rust code (rustfmt-clean) inside an impl or mod block, for example: impl Foo { const DOC: &str = r#" #[cfg(test)] mod tests { "#; pub fn evil(&self) { ... } } The text enters through collect_tests -> rust_test_ranges(head_txt).
  • Mechanism: rust_test_ranges matches #[cfg(test)] and mod tests { line by line with regexes, even inside string literals and comments, and then counts braces. The fix that is supposed to reject over-long ranges requires only two things: the last line is a } at the mod line's indentation, and every inner line is indented deeper. In the example the mod line has indentation 0, the string's closing line and all impl members are indented 4 or more, and the impl's closing } sits at column 0. So the range [3, 9] is accepted. Comment braces such as // { and { // } also let the counted range run past the real end of a module when the code is not rustfmt-formatted. strip_test_modules then drops every +/context line in that range and seed_diff hands the result to the finders. The existing tests only cover unbalanced braces whose real close is at the module's indentation.
  • Impact: Production code, such as a new pub fn in an impl, is replaced in every finder's seed diff by '[N lines of a #[cfg(test)] module omitted; tests are listed in <tests>]' and counted as test lines at half weight. The PR description says this hiding route is closed. A malicious author can hide a backdoor from the AI reviewers unless one of them decides to open the file.
  • Judge: The raw-string example passes the implemented checks: regex matching recognizes the fake attribute/module inside the string, brace counting reaches the enclosing impl's closing brace, and interior indentation is deeper than the fake module line. Production methods in that range are consequently stripped. The indentation guard does not establish lexical Rust boundaries.

M16. Rust test stripping also removes production items on the closing line (medium, logic)

  • Trigger: A PR author changes a Rust test module's closing line from } to } pub fn authorize() -› bool { true }. Rust permits the following production item on the same line; the preceding #[cfg(test)] applies only to the module.
  • Mechanism: rust_test_ranges checks that the final line starts with a closing brace, but does not reject subsequent production tokens. The appended function's balanced braces leave the module's counted depth at zero, and the indentation check still passes. collect_tests records the entire closing line as test content, and strip_test_modules removes that line from the seed diff. Both seed_diff and its ranked fallback use this stripping, while not_reviewed does not flag stripped ranges in an otherwise assigned code file.
  • Impact: Production changes disappear from the normal agentic reviewers' supplied diffs and are presented as omitted test content without a missing-coverage warning. Reviewers can still discover them through independent file reads, but security-sensitive additions can evade the intended review coverage.
  • Judge: A closing line containing a balanced production item still has the expected net brace count and starts with a closing brace, so it is accepted as test content. Stripping then removes the entire line, including the production item. This is distinct from M15's false recognition inside strings: genuine test modules also trigger it.

M17. Test collection performs quadratic work on PR-controlled files (medium, DoS/resource)

  • Trigger: A PR modifies an existing test file containing many test definitions. One changed line is enough to trigger the per-test scan of defs_at; changing many lines also triggers the nested changed-line/test-definition loop. This is PR-controlled content processed by collect_tests before any review lane starts.
  • Mechanism: For every test definition, line 463 linearly rescans defs_at from its beginning, making end calculation O(T×D) for T tests and D definitions. Lines 465-468 then compare every changed head line with every test definition, adding O(N×T) work for N changed lines, with additional linear membership checks against changed. There is no file-size, test-count, or work budget on this path.
  • Impact: A large generated suite or deliberately shaped test file can consume minutes of CPU and stall or time out the workflow before a review or useful coverage comment is produced. Large honest test suites hit the same scalability failure.
  • Judge: For an existing file, every test rescans definitions from the beginning, producing quadratic work even for one changed line. Changed lines are then compared with every test, with additional list-membership costs. This unbounded preprocessing runs before review scheduling, so sufficiently large PR-controlled suites can stall planning.

M18. Deletion-only hunks of 15+ lines are replaced by a one-line summary, and under -W the whole enclosing function goes with them (medium, logic)

  • Trigger: Any PR, honest or not, that removes 15 or more lines inside a function without adding any line to that function's hunk, for example deleting a signature/bounds/permission check block from fn process_block.
  • Mechanism: seed_diff renders with -W, so a deletion inside a function becomes one hunk holding the whole function as context plus the '-' lines. compact_deletions replaces any hunk with no '+' lines and at least DELETION_COMPACT_MIN (15) '-' lines with only its @@ header and '[deletion-only hunk: N removed lines not shown; removed definitions: ...]'. Both the removed code and the surviving function body are dropped. build_plan then records these lines as compacted, so they also leave the effective size.
  • Impact: Removing validation is one of the most important things a security review should catch, and the finders never see the removed lines or the function they came from. The file only appears in the prompt's 'Files or hunks not shown in full' list, so catching the problem depends on a model choosing to run git diff itself.
  • Judge: The compactor replaces the entire qualifying hunk, not merely its removed lines. With whole-function context, a deletion-only validation removal can therefore hide both the deleted checks and surviving function body. The omission is disclosed, but inspection requires an additional tool read; disclosure does not restore the supplied review content.

M20. Split PR test files can be placed in an unreviewed part and omitted from warnings (medium, logic)

  • Trigger: A split PR contains test files whose directory has no common prefix with any code file, for example code under src/ and tests under top-level tests/. plan_units only enters this path when the total effective size exceeds NORMAL_MAX.
  • Mechanism: best_unit_from returns no unit when _common_dir is empty, so plan_units assigns those tests to u00. In run_pilot.build_plan, every unit with code_files == 0 is set to not reviewed and receives no lanes. Test files are deliberately not included in the seed diff, so no finder or housekeeping lane receives their test changes. not_reviewed only reports code or test records with an empty unit; these records have u00, and the PR comment's partial-review count filters to units with code_files, excluding u00.
  • Impact: Changed test code, fixtures, or test-only security behavior can be silently omitted from every reviewer in a split PR, while the comment neither counts the part as partially reviewed nor lists its files under Not reviewed.
  • Judge: Tests without a shared code-directory prefix are assigned to u00, which has no code and receives no lanes or test block. Their nonempty unit suppresses the file warning, while the partial-review count excludes test-only units. The PR map can list their names, but that does not schedule review of their changes.

M8. Claude Code reviewer lanes cannot run any Bash command on the Linux CI runner (bubblewrap fails to create a mount point inside the read-only checkout) (medium, logic)

  • Trigger: Any Claude Code lane run by the CI workflow on Linux: the Opus finder lane, or the judge with --judge claude. No attacker is needed; it happens on every run.
  • Mechanism: claude_settings() makes the lane's working directory (the isolated checkout wt) write-denied. Claude Code's bubblewrap sandbox also adds protections inside cwd for paths that do not exist (the deny-within-allow list covers paths such as ‹wt›/.claude/settings.json). For each of those, bubblewrap must create a mount point inside the now read-only bind of wt, and that fails. The whole sandboxed command then fails before it runs. In this very run (the run dir is yetanotherco_actions-24-…/units/u04/agentic/claude, with the AI_REVIEW_DENY paths in its deny list), every Bash call failed with bwrap: Can't create file at /home/runner/.cache/ai-review/isolated/yetanotherco_actions-24-20261117-184552/scripts: Read-only file system, while Read, Grep and Glob worked. failIfUnavailable: True does not catch this, because the sandbox reports itself as available. existing() filters only denyRead paths, not Claude Code's own write protections.
  • Impact: The Opus reviewer, about 40% of each review's cost, runs without a shell: no git diff or git log against the merge-base and no other commands. Each failed Bash call still uses up its tool budget. The lane still ends with status "ok", so nothing in the comment or the unit status shows it was degraded. The PR's first-run checklist checks git only in the Codex artifacts.
  • Judge: The runner denies writes to Claude's working checkout, and individual Bash failures would not necessarily fail the lane. However, the asserted bubblewrap mount ordering, missing mount points, and universal Linux Bash failure depend on external sandbox behavior and runtime artifacts that could not be verified. The available code supports plausibility, not confirmation.

M19. Generated chunk labels can overwrite real package areas (low, logic)

  • Trigger: A large PR has package roots pkg and pkg/{a.rs}, modifies pkg/a.rs by 2,600 effective lines and pkg/b.rs by 2,000, and changes a small source file under pkg/{a.rs}/. PR-controlled directory names and package manifests can create this layout.
  • Mechanism: plan_units uses the same dictionary namespace for real area paths and generated chunk labels. The direct files in pkg remain an oversized area after directory splitting, so chunking generates pkg/{a.rs} for the first singleton chunk. Processing the real package area pkg/{a.rs} subsequently executes this assignment and replaces that chunk. The relative-path label fix prevents collisions between ordinary chunks, but not collisions with actual area names. not_reviewed reports the now-unassigned file; it does not restore it to a unit.
  • Impact: pkg/a.rs receives no review unit or reviewers despite being ordinary reviewable code. The comment warns that it was not placed in a part, but increasing the budget or continuing the review cannot schedule it because replanning reproduces the collision.
  • Judge: Real area paths and synthetic brace-wrapped chunk labels share one dictionary namespace. Both reported layouts can make a real area overwrite a generated chunk, leaving its code unscheduled. The Not reviewed warning limits the impact, and the required unusual directory names justify low severity.

M21. Config check misses legacy keys rejected by the runner (low, logic)

  • Trigger: A future maintainer re-adds a removed configuration key such as priority or sensitive to .github/ai-review/config.toml.
  • Mechanism: The check only parses TOML and verifies that the configured threat-model file exists. It does not invoke or replicate run_pilot.py's check_config_keys, which rejects the removed keys. The offline tests therefore pass even though the next review run will refuse the repository configuration before any model review.
  • Impact: An invalid repository configuration can pass CI and merge, after which subsequent reviews fail at runtime.
  • Judge: over --max-verify 20

M22. The workflow accepts any slash word as the command without checking it against inputs.command (low, logic)

  • Trigger: A caller copies the README example, whose if is startsWith(github.event.comment.body, '/review'). Someone then comments /reviewed, /review-later or /reviewing now on a PR.
  • Mechanism: The caller's loose prefix test passes. Read options then takes the first word as the command whenever it looks like a slash command and never compares it with $COMMAND, so nothing rejects the request. The paid review runs, and --continue-command is built from the wrong word (for example /reviewed continue).
  • Impact: Comments that were not meant as review requests start paid reviews (about $3, or up to $50 for a split PR). The partial-review hint in the PR comment then tells users to type a command that the caller may not trigger on.
  • Judge: over --max-verify 20

M23. Spaced lane lists skip the Anthropic secret check (low, logic)

  • Trigger: A caller passes the documented comma-separated lanes value with a conventional space after a comma, such as lanes: 'gpt, claude'.
  • Mechanism: For gpt, claude, the padded string is ,gpt, claude,; the pattern *,claude,* does not match because a space precedes claude. The workflow therefore skips its ANTHROPIC_API_KEY check, but later passes the lane list unchanged to run_pilot.py, which strips spaces, accepts claude, requires the key in run_paid, and exits before reviewing. I reproduced the shell condition locally.
  • Impact: A valid lane configuration fails after setup with an Anthropic-key error instead of failing fast, wasting a workflow run and producing no review.
  • Judge: over --max-verify 20

M26. An infinite line value in a finding raises OverflowError and discards all of that reviewer's findings for the part (low, logic)

  • Trigger: Reviewer output with "line": "inf", "Infinity", 1e999 or "1e400". Non-OpenAI lanes parse the schema leniently, and PR text could prompt-inject such a value.
  • Mechanism: float(...) returns inf, and int(inf) raises OverflowError, not ValueError, so the coercion's except ValueError does not catch it. The exception reaches parse_lane_findings's except Exception, which marks the lane failed and keeps no findings.
  • Impact: One malformed field drops every finding that lane reported for the unit, instead of only setting that finding's line to 0. The part is then shown as partially reviewed.
  • Judge: over --max-verify 20

M27. One housekeeping item with a non-integer line (e.g. "45-60") discards the whole housekeeping lane (low, logic)

  • Trigger: An ordinary model answer. The housekeeping lane runs DeepSeek flash through Codex, and its schema is not enforced (SCHEMA_ENFORCED_PREFIXES is openai/ only). An item with "line": "45-60", "README.md:12" or "L12" triggers it.
  • Mechanism: int("45-60") raises ValueError inside the loop. The surrounding except Exception then marks the whole lane failed ('output not parseable as housekeeping JSON') and drops every item already parsed. normalize_findings handles the same input leniently (split on '-', fallback 0), but this path does not.
  • Impact: All housekeeping notes for that part are lost because of one malformed line number, and the comment's housekeeping section is missing or empty. The lane's spend is still counted.
  • Judge: over --max-verify 20

M29. After a cancelled or aborted run, parts not yet finished keep status "planned" and are left out of the comment's partial-review list and continue hint (low, logic)

  • Trigger: A run interrupted during the unit loop: the job is cancelled or hits the 330-minute timeout (SIGINT/SIGTERM become KeyboardInterrupt), or a runner exception occurs while a split PR's parts are running.
  • Mechanism: Unit statuses are set only inside the per-unit loop in run_paid: "not reviewed (…)" before a unit runs, and "reviewed"/"partially…" after its phase returns. On an exception, except BaseException only records crash and trips GUARD. The in-progress unit and every later unit keep the seed default status = "planned". pr_comment lists only statuses starting with "not reviewed"/"partially reviewed", so those parts are omitted.
  • Impact: The comment says the review stopped early but does not name the parts nobody reviewed, and does not show the /review continue hint for them. A reader can take the shown findings as covering the whole PR. That goes against the PR's claim that a cancelled run still writes a comment listing what was not reviewed.
  • Judge: over --max-verify 20

M30. Zero PR budget disables all review instead of disabling the budget (low, logic)

  • Trigger: A local operator runs --pr-budget 0 without an explicit --run-cap, following the option help that says 0 disables the per-PR budget.
  • Mechanism: --pr-budget 0 is passed through to args.run_cap. The unit gate then computes unit_limit = min(0, 0) - verify_reserve(...), which is negative, so every unit with fresh lanes is skipped as over budget. The option therefore does not disable budget gating; it makes the effective limit impossible to satisfy.
  • Impact: An honest local review performs no finder lanes and reports every part as not reviewed by budget.
  • Judge: over --max-verify 20

M31. Cancelled runs omit interrupted and unstarted parts from coverage warnings (low, logic)

  • Trigger: SIGTERM, Ctrl-C, or another exception interrupts a paid run while a unit's finder phase is in progress, including CI cancellation.
  • Mechanism: The exception jumps past the per-unit status assignment at lines 4479-4489. Crash recovery restores biller rows but never changes the active unit, or later units, from their default status of planned. pr_comment only includes statuses beginning with not reviewed or partially reviewed in its partial-coverage list.
  • Impact: The comment contains only the generic stopped-early warning and omits which parts were interrupted or never started, along with the continue hint. On a split PR, users cannot determine the missing coverage from the posted comment.
  • Judge: over --max-verify 20

M32. CODEOWNERS and dependency .txt files are classified as docs and kept out of the security review (low, logic)

  • Trigger: A PR edits .github/CODEOWNERS, for example removing the required owners of sensitive paths, or requirements/prod.txt / deps/*.txt (pip requirements kept in a directory). NOT_DOCS only exempts file names that start with requirements/constraints.
  • Mechanism: classify_files gives kind 'docs' to names in DOC_NAMES and to .txt files not matched by NOT_DOCS. Docs go only into docs_diff, which the housekeeping lane sees. unit_values gives the finders seed_diff only, and not_reviewed() does not list docs.
  • Impact: Changes to access-control config (who must approve which paths) and to Python dependency pins are seen only by the cheap stale-docs lane. They get no security review, and the comment does not say so.
  • Judge: over --max-verify 20

M33. Glob matcher does not support standard character classes (low, logic)

  • Trigger: A repository's base-branch config.toml uses a standard fnmatch-style character class in ignore, always_review, low_risk, test_globs, or --paths, such as foo/[bt]ar.rs.
  • Mechanism: The implementation explicitly replaces fnmatch to prevent * crossing /, but its translator handles only **, *, and ?; every other character, including [, !, and ], is escaped literally. I confirmed glob_match("foo/bar.rs", ["foo/[bt]ar.rs"]) and glob_match("foo/tar.rs", ["foo/[bt]ar.rs"]) both return None. There is no alternate fnmatch path in classify_files.
  • Impact: Repository classification rules silently fail to match their intended files. A failed always_review expression can under-review behavior-driving files, while a failed ignore expression causes unintended review work; failed low_risk rules fall back safely to full review.
  • Judge: over --max-verify 20

M34. Renamed/copied deletion compaction is attributed to the old path, leaving compaction/effective-size totals wrong (low, logic)

  • Trigger: Any rename or copy whose diff contains a large deletion-only hunk (at least DELETION_COMPACT_MIN lines) that gets compacted.
  • Mechanism: For rename/copy diff headers, --- a/old appears before +++ b/new, so re.search returns the old path. compact_deletions then puts the saved compaction count under old_path, but classify_files keys records by new_path. In run_pilot.build_plan, by_path[p] looks up the new path, so the renamed/copied FileRec never gets its compacted field updated. Later sizes() and the recalculated unit effective lines therefore omit those compacted deleted lines.
  • Impact: The reported compacted-deleted-line count and effective PR/unit size are incorrect for renamed or copied files with deletion-only hunks; review allocation itself is unaffected.
  • Judge: over --max-verify 20

M35. strip_test_modules drops a file's trailing newline when its last hunk ends inside a test module, gluing the next file's diff header onto the marker line (low, logic)

  • Trigger: A Rust file whose last hunk ends before the end of its #[cfg(test)] module. This is common: under -W, editing any test function except the last one stops the hunk at the next fn, and -U20/-U3 hunks stop even earlier. Another file must follow it in the diff.
  • Mechanism: Each per-file chunk ends with '\n', so ch.split('\n') ends with an empty string. That element falls into the context branch with n set to one past the last hunk line, which is still inside the head test range, so it is dropped and counted (the marker overstates the count by one). The function then appends the marker with nothing after it, and '\n'.join(res) has no trailing newline. The next chunk's 'diff --git' is concatenated onto the marker line. compact_deletions splits on '^diff --git ', so it then treats the next file's header and hunks as hunks of the previous file. Its deletion-only compaction counts are recorded under the wrong path, and build_plan stores them in that file's compacted.
  • Impact: The seed diff is malformed (a file header in the middle of a line). The next file's compacted line count is added to the previous file, which shrinks that file's effective size and can skew unit sizing. The off-by-one count is cosmetic.
  • Judge: over --max-verify 20

M36. Test-only PRs create units that can never be reviewed (low, logic)

  • Trigger: An ordinary PR changes only files matched by the built-in or configured test globs, with no production code file. plan_units includes those records in reviewable and takes the normal-size path.
  • Mechanism: The returned unit has code_files == 0, but every test record is assigned r.unit = "u01". In run_pilot.py:3052-3053, the zero-code unit is marked not reviewed and its lanes are cleared; run_pilot.py:3061-3073 consequently builds neither a diff nor the test listing. The coverage check at seed.py:931 only reports code/test records with no unit, so these assigned records are omitted from not_reviewed, and the comment's skipped-part calculation filters out zero-code units. Re-running or using continue follows the same path.
  • Impact: Test additions, removals, and weakened assertions receive no reviewer at all, contrary to the documented behavior that reviewers receive changed test files and test functions. The comment does not identify the skipped test files or offer a working continuation path.
  • Judge: over --max-verify 20

M24. Re-running a failed review job may fail to upload its artifact, so the old review is posted (low, logic)

  • Trigger: A review job fails (for example run_pilot exits 3 for a killed run, or crashes after writing its report), and a user clicks "Re-run failed jobs" for the same PR head.
  • Mechanism: The artifact name ai-review-‹PR›-‹head sha› is the same for every attempt of a workflow run, and the upload step does not set overwrite: true. upload-artifact v4 and later refuse a second artifact with the same name in the same run. The re-run attempt pays for a full review, its upload fails, and the post job then downloads the earlier attempt's artifact by that name.
  • Impact: The re-run spends the full review budget again, but the PR gets the old comment (with a "did not finish cleanly" note). A later /review continue also reuses the old artifact instead of the new results.
  • Judge: over --max-verify 20

M25. Cost of harness lanes routed via a static preset is estimated from a dynamically chosen host that may not be in the preset (low, logic)

  • Trigger: Prices or endpoints change on OpenRouter after the presets were created (2026-11-11/12). Or the first host in a preset's order (e.g. baidu) is unavailable and requests fall through to later hosts in the order.
  • Mechanism: The DeepSeek and GLM presets hold a fixed provider order. Routing.price_map(harness=True) prices these lanes at eps[0] of pinned_endpoints(), which is recomputed on each run from LAB_PROVIDERS (cheapest cached-read fp8 host under the cap). harness_model only checks that a preset exists and provider() is non-None. It never compares the preset's stored order with today's allowed hosts, so the host used for the price may not be one the preset routes to.
  • Impact: The per-review budget gate and the cost line in the comment can understate spend for the DeepSeek and GLM lanes. Only the OpenRouter key's credit limit remains as a hard ceiling.
  • Judge: over --max-verify 20

M28. Comment sanitizer's code/text parity tracking can be desynchronized with double-backtick spans, letting &#64;user entities render as mentions (low, injection)

  • Trigger: Model-written text (finding title, trigger, mechanism, judge reason, housekeeping text). A PR author can steer it with text in the PR, or a model can produce it on its own, e.g. a title like "a`b@octocat`".
  • Mechanism: _md_text splits on every single backtick and assumes odd segments are code spans. Only text-parity segments get & escaped; code-parity segments only get ‹/› replaced. CommonMark matches backtick runs by length, though. In "a`b@octocat" the split gives 7 parts, and "&amp;#64;octocat" lands at index 5 (code parity), so its & is not escaped. GitHub renders "``ab``" as one code span, the entity as text ("@​octocat"), and the final backtick as a literal. The @ regex does not fire because there is no literal '@'. The test checks entities only in out.split("")[0]`, so this case is not covered.
  • Impact: The bot's PR comment can contain a rendered @​user or @​org/team mention, which the docstring says must not happen ("also not as entities such as &#64;"). That can notify arbitrary users or teams. No HTML, links or images get through, because ‹/› are replaced in both parities.
  • Judge: over --max-verify 20
Duplicates folded into another finding: 2
# Same as Finding Found by Where
M12 M3 git diff -w hides behavior-changing whitespace edits in string literals and in whitespace-sensitive files not on the W Opus ai-review/seed.py:61
M13 M3 Semantic string edits are discarded as whitespace-only Kimi ai-review/seed.py:245
Pre-existing (real, but already on the base branch): 1
# Finding Category Severity (reported → judged) Found by Where Judge's reason
M6 Authorized users can repeatedly trigger unbudgeted total spend funds-loss medium → low GLM (1/5) .github/workflows/pr_review_multi.yml:133 Repeated authorized requests could already consume shared provider credits on the base branch: /review invoked paid single-model workflows without a cumulative quota. The new runner likewise limits individual runs rather than aggregate spending, but does not introduce that underlying capability. The new triage authorization path is covered by M5. The claim that every pending request queues indefin
Housekeeping (not security): 2

Stale docs, missing tests, leftovers and inconsistencies, from a separate cheap model. Not checked by the judge.

Kind Item Where Fix
stale-docs ANTHROPIC_API_KEY secret description omits fable lane .github/workflows/pr_review_multi.yml:60 Change the description to: 'Anthropic API key for the Claude Code reviewers (claude/fable); required unless lanes leaves out all Claude Code lanes'
stale-docs Secrets check error message only names Claude when Fable also needs the key .github/workflows/pr_review_multi.yml:154 Change the error message to: 'the ANTHROPIC_API_KEY secret is empty, and the reviewers include Claude or Fable'

Cost of this review: ~$16.96 of a $50 budget (estimated from token counts).

One review covers the whole PR. A PR over 20,000 effective lines is not
reviewed: the comment gives its size and asks for it to be split. The
splitting, part ordering, part map and per-part budget gate are gone;
the default budget is one $30 per PR.

On propamm #1 (one run per arm), one review of 10k or 21k lines found
the critical and the same real highs as four ~2.5k-line parts at about
half the cost. The prompt still holds 250k chars of diff: Codex compacts
its context well below the models' 1M windows (GLM at a 408k-char
prompt, GPT at 795k), so a bigger diff would be lost to its summary.
Above 250k the diff is ranked, and a file too big to fit is cut where
the room ends instead of being dropped. A Codex compaction is now logged
and resets the in-flight cost estimate, which kept counting the dropped
context. Reviewers' dollar limits grow with the prompt size.

Malicious-code check: one cheap call (DeepSeek V4 Flash, GPT as
fallback) over what the security reviewers do not see and what runs on
a developer's machine or in CI by itself (tests, build and install
scripts, hooks, editor tasks, dependency sources), plus the lines a
pattern scan matched anywhere. Findings go at the top of the comment.
On 10 planted attacks and 6 look-alike clean changes it caught 10 and
flagged none; on 10 real merged PRs it raised one item.

Housekeeping gets 150 commands (60 stopped it on a 5-line part).
In the first CI review (this PR, run 44151243197) four Codex lanes
reached their soft limit and not one command was refused: DeepSeek ran
50 commands past its stop and was killed at the 35-minute timeout with
no answer, twice. The gate's zsh files lived in the lane folder under
the runs dir, which CI read-denies, with a more specific "read" entry
for that folder. macOS honours the nested entry; Codex's Linux sandbox
does not, so zsh could not read the gate.

The gate files now live in ~/.ai-review-gates/<hash of the lane folder>,
outside every denied path. The sandbox self-check also closes a gate
and runs a zsh command through the lane sandbox: it logs whether the
command was refused, so the next CI run shows the gate working on
Linux before any paid call. Checked on macOS: the probe is refused.
It read only tests, files that run by themselves and files with a
pattern hit, so a harmful helper in product code, called from an
innocent-looking test line, was out of its view. It now reads every
changed file but docs and lockfiles, files with a hit first, then tests
and files that run by themselves, then the rest, up to 600k chars (one
direct call: Codex's compaction does not apply; about a cent).

Checked with variants written to avoid every pattern (paths and hosts
built from pieces, hex-encoded commands, a `pretest` hook, Make
variables, a product-code helper called from a test), planted into a
real 5,200-line waas PR: 6 of 6 caught at high confidence, twice. The
16 corpus cases still score 10 of 10 caught, 0 of 6 flagged, and 10
real merged PRs get no item ($0.002-0.011 each).
…er compact the diff away

Codex 0.142.2 knows none of the lanes' OpenRouter model strings. On its
fallback metadata it caps the context window at 272k and compacts at
244,800 tokens whatever config.toml says, and compaction replaces the
diff with a summary. Every Codex lane (finders, judge, housekeeping) now
gets a one-entry model catalog (model_catalog_json) for the exact model
string it sends, preset suffix included, with the model's OpenRouter
context length as its window. Compaction is at 0.85 of it, and at most
0.97 of the hosts' max prompt where OpenRouter gives one (GPT-6.1-sol:
922k).

The entry (ai-review/codex_catalog_entry.json) is a frozen copy of
Codex 0.142.2's fallback metadata, base instructions included, so the
requests stay the same; it must be re-captured when the pinned Codex
version changes. In the pilot: 0 compactions, GPT up to 284k and GLM up
to 345k tokens per request.

Each lane also writes requests.jsonl (per-request token usage, Codex's
effective window, compaction markers), and the runner logs a WARNING if
a lane still compacts.

With the compaction gone, the diff in the prompt goes from 250k to 600k
chars (about 15,000 lines of whole functions, 150-250k tokens).
Its 262k-token window cannot hold a full 600k-char diff plus room to
work; the default Kimi reviewer is K3.1 Pro (1M). Nobody selected it.
Compaction is left on as a backstop, but every one is now counted, so
the prompt size can be tuned from real runs: per agentic lane (Codex and
Claude Code, finders, judge and housekeeping) its compactions, its
largest single request in tokens and its window, in summary.json, a
"Context" section of the report and one line of the job log. Not in
the PR comment.
…budget

The script from the previous commit (missing paths and make targets,
new names nothing references) is no longer meant for the housekeeping
prompt. Measured on propamm #1 and actions#24 (one or two samples per
variant): with the leads in its prompt the model only checked leads and
stopped searching on its own, and an added "then search on your own"
line did not change that. Without them, the old prompt found a real
issue no lead covers. So the model keeps the old, short prompt, and the
leads go to their own collapsed section of the comment, marked as not
checked: on propamm #1 that is 17 leads, 16 real, including 2 that no
model run found.

Housekeeping's dollar limits now grow with the prompt size, as the
finders' do: on a 711k-char prompt it stopped after ~38 of its 150
commands.
@MauroToscano MauroToscano added ai-review Run the multi-model AI review on this PR and removed ai-review Run the multi-model AI review on this PR labels Nov 18, 2026
The cap was 20 findings, set when each finding had its own Claude
session (~$0.39 each). The first CI review of actions#24 (36 findings)
left 16 lows unjudged. The judge is now one GPT session (~$0.30), and
15 measured sessions of 3-14 findings used 9-22 commands and 2-5
minutes, about 1-2 commands per finding. So: cap 60, and the session's
soft limit is 15 + 2 commands per finding (up to 150), 5 minutes + 20 s
per finding (up to 25). One session also keeps duplicate detection
across all findings.
@github-actions

Copy link
Copy Markdown

AI security review: 16 finding(s) to check

Reviewed by Opus, GPT, Kimi, GLM. A judge model checked every one of the findings against the code.
Partial review: this PR was not fully reviewed (reviewers failed). Comment /review continue to finish it, with a new budget; reviewers that finished are reused.
Malicious-code check (tests, build and install scripts, hooks, editor tasks, dependency sources): nothing found.

# Finding Severity Status Found by Where
M1 Label-triggered reviews execute PR-controlled workflows with API secrets high Bug Kimi (1/4) .github/workflows/ai-review.yml:40
M2 Reviewer sandbox omits common local credential stores high Bug GLM (1/4) ai-review/run_pilot.py:1338
M3 Claude Bash path is not covered by the /proc deny high Suspected GLM (1/4) ai-review/run_pilot.py:1690
M15 Do not silently omit oversized files from the tool-less malicious check medium Bug GLM, GPT, Kimi (3/4) ai-review/seed.py:871
M10 Invalidate reused reviews when the merge-base changes medium Bug GLM, GPT (2/4) ai-review/run_pilot.py:4627
M5 Codex artifacts can disclose unmasked local Git credentials medium Bug Kimi (1/4) ai-review/run_pilot.py:1642
M6 Hit truncation hides lockfile dependency sources from every reviewer medium Bug Kimi (1/4) ai-review/run_pilot.py:2144
M7 Preserve confirmed verdicts when folding judge duplicates medium Bug GPT (1/4) ai-review/run_pilot.py:3473
M8 OpenRouter preflight counts Anthropic-billed Opus spend medium Bug GLM (1/4) ai-review/run_pilot.py:3928
M9 Continue preflight does not subtract reusable reviewer cost medium Bug GLM (1/4) ai-review/run_pilot.py:4380
M11 Whitespace filtering can hide semantic string changes medium Bug GLM (1/4) ai-review/seed.py:276
M12 #[cfg(test)] followed by an item on the same line makes the next mod x { count as a test module and hides it from re medium Bug Opus (1/4) ai-review/seed.py:365
M13 Raw brace counting can still hide production Rust as test code medium Bug GPT (1/4) ai-review/seed.py:372
M14 Decode Git-quoted paths before scanning lockfile changes medium Bug GPT (1/4) ai-review/seed.py:833
M16 A PR with no code files gets 'No reviewer finished' and is not reported as not reviewed low Bug Opus (1/4) ai-review/run_pilot.py:3481
M17 Deletion-only hunks of 15+ lines are replaced, with their whole-function context, by a one-line note, hiding removed che low Bug Opus (1/4) ai-review/seed.py:548
Details

M1. Label-triggered reviews execute PR-controlled workflows with API secrets (high, key/secret handling)

  • Trigger: A user with write access pushes a same-repo branch that modifies ai-review.yml or pr_review_multi.yml, then adds an ai-review label (or a maintainer labels that PR).
  • Mechanism: The pull_request/labeled event runs workflow files from the PR merge commit, as the adjacent comment explicitly notes. The local reusable-workflow reference therefore selects PR-controlled code, and lines 47-49 pass it both API keys. The same-repository check only excludes forks; it does not make a same-repo branch trusted, and no environment approval or base-branch workflow pin intervenes.
  • Impact: A branch that has not passed review or branch protection can run arbitrary steps with OPENROUTER_API_KEY and ANTHROPIC_API_KEY, exfiltrating them and abusing their spend limits.
  • Judge: The label event selects the PR merge commit's local reusable workflow and passes both API keys. The reusable workflow also checks out its own workflow SHA as executable runner code. Fork rejection and base-branch configuration loading do not protect against modifications on a same-repository branch.

M2. Reviewer sandbox omits common local credential stores (high, key/secret handling)

  • Trigger: A developer runs the documented local runner on a PR containing prompt-injection content while a standard credential file such as ~/.npmrc, ~/.git-credentials, ~/.pypirc, or ~/.gem/credentials exists.
  • Mechanism: Both Codex and Claude start from this credential deny list, but these common files are absent from it. The read-only sandbox otherwise permits disk reads, so an injected reviewer can read a token and place it in model output. mask recognizes OpenRouter, Anthropic, and GitHub token forms, not npm, PyPI, RubyGems, or arbitrary basic-auth credentials, and the PR-comment sanitizer does not remove bare token text.
  • Impact: A package-registry or Git credential can be exposed in lane logs, artifacts, and the public PR comment.
  • Judge: Both sandbox configurations omit the named credential stores while permitting reads outside denied paths. Registry tokens and arbitrary Git passwords are not covered by the masking expressions. Disclosure into model requests and saved output is reachable; a local run does not itself automatically publish a PR comment.

M3. Claude Bash path is not covered by the /proc deny (high, key/secret handling)

  • Trigger: A prompt-injected Claude/Opus lane uses the allowed Bash tool to read /proc/‹claude-pid›/environ, which contains ANTHROPIC_API_KEY, on Linux.
  • Mechanism: The Read(//proc/**) permission rule applies only to Claude's Read tool, while Bash is explicitly allowed. The sandbox denyRead list contains the sensitive paths but not /proc, and no PID-namespace configuration is present in these settings. If Claude Code's own sandbox does not already isolate /proc, a Bash command can read another same-user process's environment. The runner does not run its /proc canary check for Claude lanes.
  • Impact: The Anthropic API key could be read and exfiltrated in an encoded form through model output, logs, artifacts, or the PR comment.
  • Judge: The configuration allows Bash, excludes /proc only through a Read-tool rule, and puts the Anthropic key in Claude's environment. However, the repository does not contain Claude Code's sandbox implementation, so its PID isolation or implicit /proc masking cannot be established. The runner's canary check covers Codex only.

M15. Do not silently omit oversized files from the tool-less malicious check (medium, logic)

  • Trigger: A PR includes a test-file diff larger than 600,000 characters, or enough earlier files to exhaust that allowance. This can remain below the effective-line limit; for example, 15,000 test lines count as only 7,500 effective lines.
  • Mechanism: The entire file is replaced with an instruction to read the checkout. However, malicious_check only makes a chat_call and exposes no tools, including on its fallback model. The security seed excludes test bodies, and regex hits are not a substitute for the omitted code: harmful operations assembled across lines need not match any pattern. No omitted-file metadata reaches malicious_lines or the coverage warning.
  • Impact: The sole reviewer intended to inspect these test bodies never receives them, yet a successful empty response is presented as 'Malicious-code check ... nothing found' rather than an incomplete check.
  • Judge: Over-cap files are replaced by checkout-reading instructions, but both malicious-check models receive only a direct chat request without tools. Test bodies are absent from the security seed, and successful empty output still reports nothing found. The additional 150-hit truncation claim overlaps M6; whole-file omission is a separate defect.

M10. Invalidate reused reviews when the merge-base changes (medium, logic)

  • Trigger: A PR is reviewed, then retargeted to a branch with an earlier merge-base without changing its head. The newly included changes touch the same filenames, and a user requests /review continue.
  • Mechanism: Artifact lookup is keyed by PR number and head SHA, so it restores the previous run. Reuse validation compares only the head and unit filename lists, not the saved base SHA or actual diff. Moving the merge-base backward can add previously unreviewed changes within those same files while satisfying both checks. Successful finder results are then reused, and the judge only examines findings those earlier finders produced.
  • Impact: The continued run can report a completed review while skipping newly in-scope changes and their bugs. Comparing the saved merge-base with mb would prevent this stale reuse.
  • Judge: Reuse validates the head and unit filenames but not the saved merge-base. Retargeting or changing the base can therefore expand the diff within the same files while all completed finders are reused. The later judge checks existing findings rather than replacing the skipped discovery pass.

M5. Codex artifacts can disclose unmasked local Git credentials (medium, key/secret handling)

  • Trigger: A developer runs a local review of a prompt-injected PR on a machine with ~/.git-credentials or credentials embedded in ~/.gitconfig, and the reviewer follows the injected instruction to read and quote them.
  • Mechanism: The Codex sandbox reads the whole disk by default, while LANE_READ_DENY (and the analogous Claude list) omits both standard Git paths. Codex writes its answer directly to last_message.txt; unlike stdout and stderr, that file is never passed through mask_file_in_place, and the workflow's rsync includes it in the uploaded artifact. Credentials that do not match the narrow built-in token regex also survive subsequent report/comment writes.
  • Impact: Git host usernames, passwords, or tokens can be published in the review artifact and potentially the PR comment, compromising accounts unrelated to the API-key sandbox protections.
  • Judge: Codex writes last_message.txt directly, and that file is never masked before artifact collection. This is a distinct redaction defect even for recognizable GitHub tokens; the missing Git credential-path restrictions also overlap M2. Local runs retain the unsafe file, but do not automatically upload it or post a comment.

M6. Hit truncation hides lockfile dependency sources from every reviewer (medium, logic)

  • Trigger: A PR adds at least 150 benign pattern matches in a lexically earlier development/test file, followed by a malicious git URL or tarball dependency in a lockfile.
  • Mechanism: Only the first 150 hits reach the model; the rest become a count. Independently, malicious_scan stops scanning added lines after 300 hits. Lockfiles are excluded from both the normal security-review diff and the malicious-check diff, so their pattern hits are their only inspection path. No later stage reconstructs or displays the omitted dependency lines after a successful check.
  • Impact: A malicious dependency source can be absent from every model's input while the comment still says the malicious-code check found nothing, exposing developers and CI to attacker-controlled packages.
  • Judge: Earlier benign matches can displace lockfile dependency-source hits from the 150-hit model input, or prevent their discovery at the 300-hit scan limit. Ordinary lockfiles are excluded from both review diffs, and a successful malicious check does not surface the omitted hits as incomplete coverage.

M7. Preserve confirmed verdicts when folding judge duplicates (medium, logic)

  • Trigger: The judge returns a schema-valid response marking M1 likely_non_issue and M2 bug with duplicate_of set to M1, or otherwise gives a duplicate a stronger verdict or severity than its target.
  • Mechanism: apply_verdict accepts duplicate_of independently of the verdict. pr_comment unions only the reporting lanes, then removes the duplicate before constructing the main findings table. It neither reconciles conflicting verdicts nor preserves the strongest severity. Thus M2's confirmed Bug verdict is discarded in favor of M1's dismissal; even the guard against dismissing a finding reported by both GPT and Claude does not protect this folding path.
  • Impact: A judge-confirmed bug can disappear from the actionable table and leave the comment saying 'No findings to check'. The duplicates section retains only its title and location, not the conflicting confirmed verdict.
  • Judge: duplicate_of is accepted independently of verdict and severity. Comment rendering unions reporting lanes but removes the duplicate without reconciling its stronger result. A confirmed bug can therefore be folded into a dismissed finding, leaving no actionable finding and hiding the conflicting verdict in the duplicate listing.

M8. OpenRouter preflight counts Anthropic-billed Opus spend (medium, logic)

  • Trigger: A CI review runs with the workflow's --claude-auth anthropic while the OpenRouter key has less remaining credit than the full five-reviewer estimate, but enough for the OpenRouter-only lanes; the separate Anthropic key is funded.
  • Mechanism: Only claude_auth == "subscription" excludes Claude lanes from tot_mode. With claude_auth == "anthropic", Opus's API-equivalent estimate is included in est_total. run_paid then compares that total only with the OpenRouter key/account remaining credit in preflight_abort, even though Anthropic pays Opus separately.
  • Impact: A review that could complete on the available OpenRouter and Anthropic balances is rejected before any paid call, posting an error instead of a review.
  • Judge: The estimate excludes Claude lanes only for subscription authentication, not Anthropic API authentication. Anthropic-billed Opus cost therefore contributes to the amount checked against OpenRouter credit, causing the stated false preflight rejection when min(estimate, run cap) exceeds that credit.

M9. Continue preflight does not subtract reusable reviewer cost (medium, logic)

  • Trigger: A budget-limited review is followed by /review continue on the same commit when the OpenRouter key's remaining credit is below the estimate for a complete fresh review but above the cost of the missing lanes and judge.
  • Mechanism: dry_run_report estimates every configured reviewer before run_paid evaluates --reuse-from. The unchanged full est_total is passed to preflight_abort, while reusable lanes are discovered only later and the actual watchdog correctly excludes reused rows because they never enter Biller.rows. No earlier check adjusts the preflight estimate for reuse.
  • Impact: Continue can abort before any paid call even though the remaining budget and key credit are sufficient for the work that still has to run.
  • Judge: Preflight uses the fresh-review estimate before reuse eligibility is inspected. Successful reused reviewers subsequently bypass job submission, so the initial estimate can reject a continuation whose missing work would fit the remaining credit. Lowering the requested budget is a workaround, not an automatic correction.

M11. Whitespace filtering can hide semantic string changes (medium, logic)

  • Trigger: A PR changes only spaces inside a string literal in a Rust, Go, TypeScript, C, or similar non-whitespace-sensitive source file, for example changing "ok" to " ok ".
  • Mechanism: For files outside WS_EXTS, classification uses git diff --numstat -w. Git ignores whitespace without understanding string syntax, so the file disappears from the whitespace-insensitive numstat and is classified as a whitespace-only change. It is excluded from the review unit and seed diff; only a configured always_review glob or a reviewer voluntarily reading an omitted file compensates.
  • Impact: A semantically important error message, protocol value, or authentication string change is omitted from the security review rather than reviewed as code.
  • Judge: Whitespace-insensitive numstat can erase changes to spaces inside string literals, and classification then excludes the file from the security seed. The comment does warn that whitespace-only files were not reviewed. Contrary to the finding's suggested compensation, always_review does not override this whitespace filter.

M12. #[cfg(test)] followed by an item on the same line makes the next mod x { count as a test module and hides it from reviewers (medium, logic)

  • Trigger: A PR author (an org member or collaborator whose same-repo PR is reviewed) changes a Rust file with a line such as #[cfg(test)] use super::*; or #[cfg(test)] #[macro_use] extern crate quickcheck; directly above a production pub mod evil {. It can also happen by accident with that common idiom. The input enters through classify_files -> collect_tests -> rust_test_ranges, called from build_plan.
  • Mechanism: _CFG_TEST is anchored only at the start of the line (^\s*#\[cfg\(\s*test\s*\)\]), so it matches a line where #[cfg(test)] already applies to an item on that same line. The code then takes the next line, after skipping any line that starts with #[, even one like #[allow(..)] const X: u8 = 0;. If that line opens a module (_MOD_OPEN), the whole module is recorded as a #[cfg(test)] range. The safety check added for this PR (closing } at the mod indentation, body indented deeper) passes for any normal module. strip_test_modules then removes every added, removed and context line of that production module from the seed diff and puts '[N lines of a #[cfg(test)] module omitted; tests are listed in <tests>]' in their place. The lines also count as inline_test, at half weight in effective.
  • Impact: Production code in that module never appears in the diff the reviewers and the judge get, and the marker tells them it is test code, so they have no reason to open it. A malicious change, such as a backdoor in a verification function, can be hidden this way. The PR says it closed exactly this gap ('a Rust test module can no longer hide code from the reviewers'). Honest PRs that use the one-line idiom also lose review of the following module.
  • Judge: The cfg(test) expression accepts trailing same-line items, but range detection still searches subsequent lines for a module. A following production module with normal indentation is accepted and stripped. The indentation safeguard does not establish that the attribute actually belongs to that module.

M13. Raw brace counting can still hide production Rust as test code (medium, logic)

  • Trigger: A PR author adds a Rust source file containing valid code such as: mod outer {\n #[cfg(test)]\n mod tests {\n const MARK: &str = "{";\n }\n pub fn authorize() -> bool { true }\n }
  • Mechanism: The brace inside the string increases the computed depth, so the actual test-module closing brace does not end the range. Rust permits the production function and closing braces to have the indentation shown: every intermediate line is deeper than the test module, and the outer module's closing brace matches its indentation. Consequently, the indentation checks also pass. collect_tests records the production function inside test_ranges, and every seed_diff rendering path calls strip_test_modules with that range.
  • Impact: Production changes disappear from every security reviewer's supplied diff and are undercounted as test lines, without being listed as unreviewed. For a product-only source change without pattern hits, the malicious-code check does not provide an alternate review.
  • Judge: Raw brace counting includes braces in strings and comments. In the supplied nested-module example, the string brace keeps the range open through the production function, while the chosen indentation satisfies both safeguards. That function is consequently stripped; a product-only change without pattern hits has no malicious-check fallback.

M14. Decode Git-quoted paths before scanning lockfile changes (medium, supply chain)

  • Trigger: A PR changes a dependency source in a lockfile under a non-ASCII directory, such as café/package-lock.json, with Git's default core.quotePath setting.
  • Mechanism: Git emits the patch header as a C-quoted path beginning with +++ "b/, rather than +++ b/. This parser sets path to None and skips every added line in that file. classify_files correctly recognizes the lockfile, but malicious_input deliberately excludes lockfiles from its full diff, relying exclusively on these pattern hits. not_reviewed also does not identify ordinary non-ASCII names as unusual.
  • Impact: A malicious dependency URL in such a lockfile reaches neither the security reviewers nor the malicious-code checker, and no coverage warning identifies the omission.
  • Judge: The scan recognizes only unquoted +++ b/ headers. Git's default quoting of a non-ASCII directory makes path None, so its lockfile additions yield no hits. Classification handles the filename through NUL-delimited output, but the full malicious diff excludes the lockfile and the unusual-name warning does not cover ordinary Unicode.

M16. A PR with no code files gets 'No reviewer finished' and is not reported as not reviewed (low, logic)

  • Trigger: An ordinary /review or ai-review label on a PR whose changed files are all tests, docs, or filtered files, such as a test-only PR.
  • Mechanism: plan_units puts every code and test file in one unit. build_plan sets u.status = "not reviewed", u.note = "no code in this part" and u.lanes = [] when code_files == 0 (line 3199), so no finder or housekeeping lane runs. pr_comment then keeps only units that have code_files (line 3490), so this not-reviewed status never reaches the comment. With no lane rows, ran is empty and the comment prints 'No reviewer finished.' under '0 finding(s) to check'. not_reviewed() does not list the test files either, because they were placed in u01.
  • Impact: The comment reads like a failed review rather than a skipped one, and it never says that the changed test files got no security review. A reader may assume the PR was checked.
  • Judge: A test-only or docs-only PR receives no finder lanes, but its zero-code unit is excluded from the comment's skipped-unit warning. Assigned test files also evade not_reviewed's unassigned-file condition. The comment consequently says no reviewer finished instead of explaining the deliberate security-review skip; the separate malicious check may still run.

M17. Deletion-only hunks of 15+ lines are replaced, with their whole-function context, by a one-line note, hiding removed checks (low, logic)

  • Trigger: Any PR that deletes a block of 15 or more lines inside an existing function without adding a line to that function, for example removing a signature, bounds or authorization check. This is ordinary PR content, rendered through seed_diff -> compact_deletions.
  • Mechanism: seed_diff first renders a -W diff, so the hunk holds the entire enclosing function as context. compact_deletions treats any hunk with - lines and no + lines as deletion-only and replaces the whole hunk, context included, with the @@ header plus '[deletion-only hunk: N removed lines not shown]'. Removed definitions are named only when the deleted lines contain definitions. The removed lines also go into compacted and are subtracted from the effective size.
  • Impact: The prompt's diff does not show the code whose removal is the change, nor the function it was removed from. Removed safety checks are a high-value bug class. Agentic reviewers see the file under 'deletion-only hunks compacted' and must recreate the diff themselves. The static-seed mode never sees it, and the judge gets the same compacted seed.
  • Judge: A deletion-only whole-function hunk with at least 15 removed lines is replaced wholesale, including its surrounding context. Removed checks without definitions receive only a count. Agentic lanes are told which files were compacted and can reconstruct the diff, but static-seed reviewers cannot, and the judge receives the same compacted seed.
Pre-existing (real, but already on the base branch): 1
# Finding Category Severity (reported → judged) Found by Where Judge's reason
M4 Read-only members and collaborators can authorize paid reviews authz/authn medium Kimi (1/4) .github/workflows/ai-review.yml:30 Association-based authorization without a repository-permission lookup already existed on the base branch: /review accepted these same associations and launched paid OpenRouter reviewers. The new budget option changes the spending controls, but does not introduce the underlying authorization weakness.
Housekeeping (not security): 1

Stale docs, missing tests, leftovers and inconsistencies, from a separate cheap model. Not checked by the judge.

Kind Item Where Fix
leftover Unused PROTECTED_CATEGORIES constant ai-review/run_pilot.py:179 Remove the PROTECTED_CATEGORIES constant and its two-line explanatory comment (lines 177-179).
Leads from a script (not checked): 4

Found without a model, from the lines this PR adds. Some can be wrong, e.g. a path created at runtime or a name used through reflection.

  • .comment.md does not exist in the repository (referenced at .github/workflows/pr_review_multi.yml:329)
  • review/comment.md does not exist in the repository (referenced at .github/workflows/pr_review_multi.yml:385, .github/workflows/pr_review_multi.yml:387, .github/workflows/pr_review_multi.yml:388)
  • owner/repo/ does not exist in the repository (referenced at ai-review/seed.py:1011)
  • PROTECTED_CATEGORIES is added by this PR and referenced nowhere else (ai-review/run_pilot.py:179)

Cost of this review: ~$8.49 of a $30 budget (estimated from token counts).

…its answer file; deny registry and git credentials

A lane whose last request ran out of output tokens while reasoning (DeepSeek, CI run 44227584384)
ended with an empty answer and failed. It now gets one 'codex exec resume' in the same session that
asks only for the JSON answer. last_message.txt is masked like stdout and stderr. ~/.npmrc,
~/.yarnrc.yml, ~/.git-credentials, ~/.pypirc and ~/.gem/credentials are denied to Codex and Claude
lanes (one list). Removes the unused PROTECTED_CATEGORIES.
…ext module as test code

'#[cfg(test)] use super::*;' above a production 'pub mod x {' hid that module from the reviewers.
The attribute now counts only when it is alone on its line, or followed by attributes or a comment.
…hs are scanned

The malicious-code check sees lockfiles only through their pattern hits. Those hits came after earlier
files' hits and were lost past the 300-hit scan limit or the 150-hit prompt limit; they now come first
and are always kept. The scan's diff no longer C-quotes non-ASCII paths, which it skipped.
Codex logs each model response's id with RUST_LOG=feedback_tags=info (OpenRouter's generation id; checked
against a local stub: one line per response, no key or headers in them). After the lane, each id is
looked up with OpenRouter's free generation endpoint; requests.jsonl gets one row per id with its
provider and the lane notes say e.g. 'served by Baidu 20, Novita 8'. Covers finders, housekeeping and
the judge, which all run through Codex.
Codex sends no output limit, so a provider's default applied: DeepSeek V4 Pro stopped at exactly 8,000
tokens while reasoning and gave no answer (CI run 44227584384). --setup-presets now stores max_tokens
(32,000, or the model's maximum when lower) in each preset and warns if OpenRouter does not keep it; a
lane whose preset has no limit says so in its notes. The presets must be re-created to take effect.
…w kept

A finding the judge confirmed as a bug, marked duplicate_of one it dismissed, disappeared from the
table. Each group of duplicates now keeps its strongest finding (Bug, then Suspected, Pre-existing,
dismissed; then the higher severity) and lists the others as 'also'.
… same merge-base

The pre-flight check held the whole estimate against OpenRouter credit, including Opus lanes billed
by Anthropic (--claude-auth anthropic) and, on /review continue, lanes that would be reused. It now
uses the OpenRouter-billed estimate of what will run (preflight.json keeps both). A continue reuses
an earlier run only when the head and the merge-base both match: a retargeted PR has a different
diff in the same files.
A file over the check's 600k-character input was left out silently and the comment still said
'nothing found'. The comment now says the check was partial and names those files.
… PR has no code

'Partial review: this PR was not fully reviewed (reviewers failed)' appeared when one reviewer failed
and four reviewed the whole PR. The comment now says which reviewer did not finish and why in a few
words (e.g. 'DeepSeek (empty answer) did not finish; Opus, GPT, Kimi, GLM reviewed the whole PR') with
the continue hint. 'Partial review' is kept for code no reviewer finished on, and names that part.
A PR with only tests, docs or other non-code files says it got no security review, instead of
'No reviewer finished'.
A fixed limit could cut off models that work today. The provider log
shows which provider stops early, so that one can be dropped instead.
@MauroToscano
MauroToscano merged commit 1352bc9 into main Nov 18, 2026
1 check passed
@MauroToscano
MauroToscano deleted the feat/multi-model-review branch November 18, 2026 13:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai-review Run the multi-model AI review on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants