Skip to content

fix(manager): post the judge's verdict as the judge reported it - #635

Merged
TheGreatAxios merged 4 commits into
mainfrom
cl-11521-post-the-judges-verdict-as-the-judge-reported-it
Oct 26, 2026
Merged

TheGreatAxios merged 4 commits into
mainfrom
cl-11521-post-the-judges-verdict-as-the-judge-reported-it

Conversation

@TheGreatAxios

@TheGreatAxios TheGreatAxios commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Closes CL-11521

Why

The manager evaluated a judged head a second time by replaying the judge's answers. When an answer had been dropped, or the pull request had changed in the meantime, the comment said "Some rules could not run this time" and the pull request got Needs Human Review. Both the judge's result and the code review's were forgotten once nothing was waiting, so the next event on the same head started each of them again.

What changed

  • The judge reports the whole verdict it evaluated: the comment, labels, Dos and rule results.
  • When the manager has the judge's verdict for the current head and this pass's rules still need the judge, it posts and executes that verdict as reported. It does not evaluate the head again. Before posting, it checks that the verdict names this pull request, head and host.
  • If this pass's rules no longer need the judge (a failed hard rule, a draft or a closed pull request), the pass's own verdict goes up instead.
  • While CI or other host checks are still running, the manager does not start the judge. It marks the head waiting, as it does for a policy without the judge, and starts the judge on the first pass after the checks finish.
  • A decided verdict stays on its head. "Decided" means it does not wait, is not degraded, has every check confirmed and holds no Do for checks still running. A later event on the same head posts that verdict again and starts no second judge.
  • The manager does not keep an undecided or degraded verdict, a failure, a missed deadline, or any result over 256 KiB. That keeps done's body under 1 MiB. Within the pass that has the result, the judge is not started again. The next event starts it again.
  • A finished code review stays on its head the same way. A later event on that head posts the stored review with the verdict, but only while this pass's rules still let code review through. It does not start a second review.
  • The manager no longer replays the judge's answers, and the verdict detail the portal reads no longer carries them.
  • Docs: the ARCHITECTURE.md judge and manager bullets, plus the triage-manager and triage-judge READMEs.
  • User manual: unchanged. It already describes the Waiting for CI label and the verdict comment, and a person still sees the same labels and comments. The only visible difference is that the wrong "could not run" comment stops appearing.

Verified

  • bun --conditions=intx-src test packages/triage-manager packages/triage-judge packages/triage-contracts packages/triage-workflows/src: all pass.
  • The new and changed manager tests fail without this change. They check that:
    • the posted verdict is exactly the judge's, and later events on the same head post it again without a second judge;
    • a draft wins over a stored verdict;
    • a verdict naming another pull request fails the pass;
    • an oversized result is not stored;
    • while CI runs the judge waits and the head is marked waiting, and after CI passes an undecided stored verdict is judged again and its Dos run;
    • a failed or late judge or review is not kept;
    • a stored review is posted again, and is not posted once code review is off.
  • The only scoring change is a waiting branch in renderVerdict for items marked judgeSkipped, which only the manager sets. The offline triage-workflows evals pass (2), and the paid-model eval was not run.
  • The hub's manager route and state Postgres tests pass (27).
  • bun run check, bun run lint and the policy check against origin/main are clean.

@linear-code

linear-code Bot commented Oct 26, 2026

Copy link
Copy Markdown

CL-11521

The judge reports the whole verdict it evaluated, and the pass that has it for its head mirrors and executes it without evaluating the head again. The judge's reviewer and its verdict stay on the claim for that head, so a later pass on it posts the same verdict and deploys no second judge.
A later pass on the same head posts the stored review with the verdict and deploys no second review; a failed or late review stays and is not deployed again.
A stored judge verdict is posted only while the pass's rules need the judge and it names the pass's pull request, head and host. The judge waits for the host's checks and the head is marked waiting. Undecided, degraded, failed, late or oversized results are not kept, so the next event deploys the reviewer again. A stored review is posted only while code review still applies.
@TheGreatAxios
TheGreatAxios force-pushed the cl-11521-post-the-judges-verdict-as-the-judge-reported-it branch from 47c07ba to 3d7b3fb Compare October 26, 2026 18:18
@TheGreatAxios
TheGreatAxios marked this pull request as ready for review October 26, 2026 18:18
@corbits-triage

corbits-triage Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Triage: Unsure

Waiting on:

  • checks are still running: check

Some rules could not run this time. Triage will re-check this pull request.

…udge verdict without a head

A stored verdict holding or skipping a Do for the head's CI is not decided, so a CI run that changes on the same head gets a new judge. A judge verdict that names no head counts as a failed judge instead of failing the pass.
@TheGreatAxios
TheGreatAxios merged commit 61aeec1 into main Oct 26, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant