Repository navigation
fix: keep every file in the review when a split area is chunked - #25
Merged
Merged
Conversation
plan_units stored each chunk of an oversize area under a label built
from the basenames of its first and last file. Chunks with the same
pair of basenames ({mod.rs .. mod.rs} in a Rust module tree) got the
same key, so each one replaced the previous and only the last chunk
was reviewed. The other files were in no unit, and neither the comment
nor the coverage table reported them.
Label chunks with their paths below the area instead. Chunks are
disjoint runs of the sorted paths, so every chunk gets its own key.
MauroToscano
approved these changes
Nov 16, 2026
MauroToscano
left a comment
Contributor
There was a problem hiding this comment.
Checked locally:
- The repro: before, 2 of the 16 precompile files are in a unit; after, all 16, in 3 units.
- A randomized test, 3,000 file trees with repeated basenames (mod.rs, lib.rs, gas.rs), root files and package roots. Before, 484 trees lost files. After, no file is lost and none is in two units.
Unit names are display-only and truncated in the comment, so the longer labels are safe. Agreed on the follow-up: warn when a changed code file is in no unit.
This was referenced Nov 16, 2026
MauroToscano
added a commit
that referenced
this pull request
Nov 18, 2026
* feat: add a multi-model PR review workflow pr_review_multi.yml runs Opus, GPT, Kimi, GLM and DeepSeek as read-only agents on the PR, merges their findings, has a judge model check each one against the code, and posts one comment: a table of findings, collapsed duplicates, pre-existing and dismissed findings, housekeeping notes and the review's cost. Large PRs are split into parts and reviewed in priority order within a per-PR budget; '/review continue' finishes a partial review and '/review budget=N' raises the budget. The runner (ai-review/) is the pilot's orchestrator, extended for CI: keys from the environment, Claude through the Anthropic API, repository config read from the PR base branch, and the review cost in the comment. In this repository '/review' now runs the multi-model review; the single-model commands are unchanged. * fix: harden the multi-model review after a multi-agent review Runner (ai-review/run_pilot.py): - --isolate shared: with a key other reviews use at the same time, the budget gate and watchdog count this run's own computed spend (Claude lanes included while they run) and skip key-usage settles. - --deadline-min: no new part of a split PR starts after the deadline; judging is skipped past it; a cancelled run ignores further signals and writes its report and comment, which is now written first. - --sandbox-check: before any paid call, one command in the Codex lane sandbox must run, read the checkout, fail to read a denied path and fail to find a canary in other processes' environments. The runner is undumpable (prctl), and AI_REVIEW_DENY adds CI paths to the deny lists; deny entries for missing paths are dropped (bubblewrap fails on them). - The Codex profile denies scratch entries one by one: denying the whole scratch dir broke git in every Codex lane. - continue: reuse only results of the same head and the same files per part, keep only well-formed findings, reuse housekeeping, never drop a fully reused part, and estimate only the lanes that run again. - PR comment: model text cannot inject HTML, mentions, links or images; the header says how many findings the judge checked and marks the rest unverified; a stopped run says why and exits non-zero. - The repository config's threat model must be a file next to it. - Finder prompt v2.7: bugs that hurt honest use count as much as attacks. Workflow (pr_review_multi.yml): continue only takes artifacts from this repository's own, non-fork runs of the calling workflow and extracts them safely; secrets, the fork check and the runner source are checked up front with the reason in the comment; GH_TOKEN only in the steps that need it; the runner is exec'd; a cancelled or failed review still posts something useful. ai-review.yml: call this repository's workflows by local reference; the 'latest' tag they used does not exist. README: correct budgets, config semantics and cost figures. * fix: keep every file in the review when a split area is chunked (#25) plan_units stored each chunk of an oversize area under a label built from the basenames of its first and last file. Chunks with the same pair of basenames ({mod.rs .. mod.rs} in a Rust module tree) got the same key, so each one replaced the previous and only the last chunk was reviewed. The other files were in no unit, and neither the comment nor the coverage table reported them. Label chunks with their paths below the area instead. Chunks are disjoint runs of the sorted paths, so every chunk gets its own key. * fix: report failed reviewers and unreviewed files, and stop hiding changes From a second multi-agent review of this PR: - A part whose reviewers failed (key out of credit, provider outage, timeout) is reported as partially or not reviewed, with the continue hint, instead of reviewed. - A Codex reviewer that times out or fails counts its estimated cost, not $0, so the budget gate, the watchdog and the comment see the spend. - The comment lists changed files no reviewer saw: binary to git, whitespace-only changes, unusual file names, and files placed in no part. - Python, shell, YAML and Makefiles keep whitespace changes in their counts and diffs. - seed.py: a tab or newline in a file name, a top-level `(root)/` directory, an unbalanced brace in a Rust test module, and sensitive docs no longer hide files or lines from the reviewers. - Model text in the comment: carriage returns and `&` are escaped. - The runner finds seed.py under `python -I`. - Offline tests (unit planning, coverage of changed files, the comment) run in the check workflow, which now also runs on changes to the review workflows. - Docs: what the sandbox check covers, the threat model's file type, the pinning fallback, running locally; this repository's priority list no longer matches every file. * docs: bring the runner's help texts and comments up to date The module docstring described the local pilot (Keychain only, cross-model verification per finding, docs lane). Help texts pointed at files only the pilot has (README item 8, finder_parallel.md) and left out `--claude-auth anthropic`, which CI uses. Comments said Claude Code lanes have no soft budget and that the judge is Opus. The dry run's spend table left out `--isolate shared`. The README now says ANTHROPIC_API_KEY is needed for `fable` too, and the `sandbox_check` input lists `off`. * refactor: simpler review config rules Each changed file is now handled by the first rule that matches: always_review (reviewed as code), ignore, the built-in skips (lockfiles, binaries, generated and vendored code, whitespace-only changes, docs), tests, and everything else gets every reviewer. A part made only of low_risk code gets low_risk_lanes (default gpt, kimi, glm) in any PR, split or not, unless triage promotes it. Large PRs are reviewed in path order, low-risk parts last; `continue budget=N` reviews what the budget did not reach. Removed: sensitive, default_sensitive, priority (and its double budget), nonsensitive_lanes (now low_risk_lanes) and unit_depth. A config that still uses one of them is refused with what replaces it. * ci: run the multi-model review on an ai-review label too A label event runs the PR's own workflow files, so a PR that changes the review (this one) can review itself before merge. Fork PRs are refused, as for comments. * docs: the threat model is optional Most repositories don't need one. A missing threat model is an info line in the job log, and a warning only when the config names a file that is not there. A threat_model.md without a config.toml is now used too: the workflow passed no config, so the runner never looked for it. * docs: say the review uses the default prompt when there is no threat model * ci: use the org's OPENROUTER_API_KEY secret for the multi-model review * feat: one review per PR, and a malicious-code check One review covers the whole PR. A PR over 20,000 effective lines is not reviewed: the comment gives its size and asks for it to be split. The splitting, part ordering, part map and per-part budget gate are gone; the default budget is one $30 per PR. On propamm #1 (one run per arm), one review of 10k or 21k lines found the critical and the same real highs as four ~2.5k-line parts at about half the cost. The prompt still holds 250k chars of diff: Codex compacts its context well below the models' 1M windows (GLM at a 408k-char prompt, GPT at 795k), so a bigger diff would be lost to its summary. Above 250k the diff is ranked, and a file too big to fit is cut where the room ends instead of being dropped. A Codex compaction is now logged and resets the in-flight cost estimate, which kept counting the dropped context. Reviewers' dollar limits grow with the prompt size. Malicious-code check: one cheap call (DeepSeek V4 Flash, GPT as fallback) over what the security reviewers do not see and what runs on a developer's machine or in CI by itself (tests, build and install scripts, hooks, editor tasks, dependency sources), plus the lines a pattern scan matched anywhere. Findings go at the top of the comment. On 10 planted attacks and 6 look-alike clean changes it caught 10 and flagged none; on 10 real merged PRs it raised one item. Housekeeping gets 150 commands (60 stopped it on a 5-line part). * fix: the soft-budget gate never fired on Linux In the first CI review (this PR, run 44151243197) four Codex lanes reached their soft limit and not one command was refused: DeepSeek ran 50 commands past its stop and was killed at the 35-minute timeout with no answer, twice. The gate's zsh files lived in the lane folder under the runs dir, which CI read-denies, with a more specific "read" entry for that folder. macOS honours the nested entry; Codex's Linux sandbox does not, so zsh could not read the gate. The gate files now live in ~/.ai-review-gates/<hash of the lane folder>, outside every denied path. The sandbox self-check also closes a gate and runs a zsh command through the lane sandbox: it logs whether the command was refused, so the next CI run shows the gate working on Linux before any paid call. Checked on macOS: the probe is refused. * feat: the malicious-code check reads the whole diff, up to 600k chars It read only tests, files that run by themselves and files with a pattern hit, so a harmful helper in product code, called from an innocent-looking test line, was out of its view. It now reads every changed file but docs and lockfiles, files with a hit first, then tests and files that run by themselves, then the rest, up to 600k chars (one direct call: Codex's compaction does not apply; about a cent). Checked with variants written to avoid every pattern (paths and hosts built from pieces, hex-encoded commands, a `pretest` hook, Make variables, a product-code helper called from a test), planted into a real 5,200-line waas PR: 6 of 6 caught at high confidence, twice. The 16 corpus cases still score 10 of 10 caught, 0 of 6 flagged, and 10 real merged PRs get no item ($0.002-0.011 each). * fix: Codex lanes get the model's real context window, so they no longer compact the diff away Codex 0.142.2 knows none of the lanes' OpenRouter model strings. On its fallback metadata it caps the context window at 272k and compacts at 244,800 tokens whatever config.toml says, and compaction replaces the diff with a summary. Every Codex lane (finders, judge, housekeeping) now gets a one-entry model catalog (model_catalog_json) for the exact model string it sends, preset suffix included, with the model's OpenRouter context length as its window. Compaction is at 0.85 of it, and at most 0.97 of the hosts' max prompt where OpenRouter gives one (GPT-6.1-sol: 922k). The entry (ai-review/codex_catalog_entry.json) is a frozen copy of Codex 0.142.2's fallback metadata, base instructions included, so the requests stay the same; it must be re-captured when the pinned Codex version changes. In the pilot: 0 compactions, GPT up to 284k and GLM up to 345k tokens per request. Each lane also writes requests.jsonl (per-request token usage, Codex's effective window, compaction markers), and the runner logs a WARNING if a lane still compacts. With the compaction gone, the diff in the prompt goes from 250k to 600k chars (about 15,000 lines of whole functions, 150-250k tokens). * chore: drop the optional Kimi K2.7 reviewer Its 262k-token window cannot hold a full 600k-char diff plus room to work; the default Kimi reviewer is K3.1 Pro (1M). Nobody selected it. * feat: report context compactions and each lane's largest request Compaction is left on as a backstop, but every one is now counted, so the prompt size can be tuned from real runs: per agentic lane (Codex and Claude Code, finders, judge and housekeeping) its compactions, its largest single request in tokens and its window, in summary.json, a "Context" section of the report and one line of the job log. Not in the PR comment. * housekeeping: scripted hints (missing paths and make targets, unreferenced new names) fed into the prompt * feat: list the housekeeping script's leads in the comment; scale its budget The script from the previous commit (missing paths and make targets, new names nothing references) is no longer meant for the housekeeping prompt. Measured on propamm #1 and actions#24 (one or two samples per variant): with the leads in its prompt the model only checked leads and stopped searching on its own, and an added "then search on your own" line did not change that. Without them, the old prompt found a real issue no lead covers. So the model keeps the old, short prompt, and the leads go to their own collapsed section of the comment, marked as not checked: on propamm #1 that is 17 leads, 16 real, including 2 that no model run found. Housekeeping's dollar limits now grow with the prompt size, as the finders' do: on a 711k-char prompt it stopped after ~38 of its 150 commands. * feat: the judge takes up to 60 findings; its limits follow measured use The cap was 20 findings, set when each finding had its own Claude session (~$0.39 each). The first CI review of actions#24 (36 findings) left 16 lows unjudged. The judge is now one GPT session (~$0.30), and 15 measured sessions of 3-14 findings used 9-22 commands and 2-5 minutes, about 1-2 commands per finding. So: cap 60, and the session's soft limit is 15 + 2 commands per finding (up to 150), 5 minutes + 20 s per finding (up to 25). One session also keeps duplicate detection across all findings. * fix: ask a Codex lane once more when it ends without an answer; mask its answer file; deny registry and git credentials A lane whose last request ran out of output tokens while reasoning (DeepSeek, CI run 44227584384) ended with an empty answer and failed. It now gets one 'codex exec resume' in the same session that asks only for the JSON answer. last_message.txt is masked like stdout and stderr. ~/.npmrc, ~/.yarnrc.yml, ~/.git-credentials, ~/.pypirc and ~/.gem/credentials are denied to Codex and Claude lanes (one list). Removes the unused PROTECTED_CATEGORIES. * fix: #[cfg(test)] with an item on the same line no longer marks the next module as test code '#[cfg(test)] use super::*;' above a production 'pub mod x {' hid that module from the reviewers. The attribute now counts only when it is alone on its line, or followed by attributes or a comment. * fix: lockfile and dependency-source hits are never cut; non-ASCII paths are scanned The malicious-code check sees lockfiles only through their pattern hits. Those hits came after earlier files' hits and were lost past the 300-hit scan limit or the 150-hit prompt limit; they now come first and are always kept. The scan's diff no longer C-quotes non-ASCII paths, which it skipped. * feat: record which provider served each request of a Codex lane Codex logs each model response's id with RUST_LOG=feedback_tags=info (OpenRouter's generation id; checked against a local stub: one line per response, no key or headers in them). After the lane, each id is looked up with OpenRouter's free generation endpoint; requests.jsonl gets one row per id with its provider and the lane notes say e.g. 'served by Baidu 20, Novita 8'. Covers finders, housekeeping and the judge, which all run through Codex. * feat: OpenRouter presets carry a 32k output limit Codex sends no output limit, so a provider's default applied: DeepSeek V4 Pro stopped at exactly 8,000 tokens while reasoning and gave no answer (CI run 44227584384). --setup-presets now stores max_tokens (32,000, or the model's maximum when lower) in each preset and warns if OpenRouter does not keep it; a lane whose preset has no limit says so in its notes. The presets must be re-created to take effect. * fix: when the judge folds duplicates, the strongest finding is the row kept A finding the judge confirmed as a bug, marked duplicate_of one it dismissed, disappeared from the table. Each group of duplicates now keeps its strongest finding (Bug, then Suspected, Pre-existing, dismissed; then the higher severity) and lists the others as 'also'. * fix: the credit check counts only OpenRouter's share; reuse needs the same merge-base The pre-flight check held the whole estimate against OpenRouter credit, including Opus lanes billed by Anthropic (--claude-auth anthropic) and, on /review continue, lanes that would be reused. It now uses the OpenRouter-billed estimate of what will run (preflight.json keeps both). A continue reuses an earlier run only when the head and the merge-base both match: a retargeted PR has a different diff in the same files. * fix: the malicious-code check says when files were too big to read A file over the check's 600k-character input was left out silently and the comment still said 'nothing found'. The comment now says the check was partial and names those files. * fix: the comment names reviewers that did not finish, and says when a PR has no code 'Partial review: this PR was not fully reviewed (reviewers failed)' appeared when one reviewer failed and four reviewed the whole PR. The comment now says which reviewer did not finish and why in a few words (e.g. 'DeepSeek (empty answer) did not finish; Opus, GPT, Kimi, GLM reviewed the whole PR') with the continue hint. 'Partial review' is kept for code no reviewer finished on, and names that part. A PR with only tests, docs or other non-code files says it got no security review, instead of 'No reviewer finished'. * revert: no output limit in the OpenRouter presets A fixed limit could cut off models that work today. The provider log shows which provider stops early, so that one can be dropped instead. --------- Co-authored-by: Julian Arce <52429267+JuArce@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A fix for #24. When a large PR is split into parts,
plan_units(ai-review/seed.py) can drop changed files from the review without reporting it.The bug
An area still over
UNIT_MAXafter three deeper splits is cut into chunks of aboutUNIT_TARGETlines. Each chunk is stored innxt[label(chunk)], and the label is built from the basenames of the chunk's first and last file. Chunks with the same pair of basenames get the same key, so each one replaces the previous one. Only the last chunk is reviewed.The dropped files are in no unit:
build_planlooks for docs files left without a unit, but not for code files./review continuecannot recover them, because they were never in a part that could be skipped.The review looks complete.
Example: a Rust PR
An ordinary Cargo workspace, and a PR that changes 350 lines in each of 16 EVM precompile modules (5,600 effective lines):
What
plan_unitsdoes with it:NORMAL_MAX(4,000), so the PR is split. The area starts as the package root,crates/vm.UNIT_MAX(4,000), so it is split one level deeper three times:crates/vm/src, thencrates/vm/src/evm, thencrates/vm/src/evm/precompiles. All 16 files are still in that one area.crates/vm/src/evm/precompiles/{mod.rs .. mod.rs}. Each replaces the previous one, and only the last survives.Before (#24 head,
704413f):Only 2 of the 16 precompiles are reviewed.
ecrecover,modexpand all the BLS12-381 and BN254 code are skipped.After (this PR):
mod.rsmakes this easy to hit in Rust, but any chunks with matching first and last basenames collide, for example{gas.rs .. mod.rs}in modules that also have agas.rs.The fix
Label each chunk with its paths below the area instead of basenames:
Chunks are disjoint runs of the sorted paths, so their first paths differ and every chunk gets its own key. The part names in the comment also become more useful:
{blake2f/mod.rs .. bls12_map_fp2/mod.rs}instead of{mod.rs .. mod.rs}. For the(root)area the prefix is not present, and the full path is used.Testing
The script below builds the example as a real git repository, with a base and a head commit. It then runs the same planning steps as
build_plan:classify_files,collect_testsandplan_units. The outputs above come from it, run against #24's head and against this branch.Repro script
Run it with the directory that holds the
seed.pyto test:python3 -I repro.py ai-review.Not run: the full runner (
run_pilot.py --helpneeds Python 3.11+ fortomllib; the check workflow runs it on this PR).Follow-up (not in this PR)
At the end of
plan_units, fail or warn when a changed code file has no unit, so a future bug of this kind is visible instead of silent.🤖 Generated with Claude Code