Skip to content

test(chain): run the live triage chain from webhook to verdict - #642

Merged
TheGreatAxios merged 1 commit into
mainfrom
cl-11528-add-an-end-to-end-test-of-the-live-triage-chain-from-webhook
Oct 26, 2026
Merged

TheGreatAxios merged 1 commit into
mainfrom
cl-11528-add-an-end-to-end-test-of-the-live-triage-chain-from-webhook

Conversation

@TheGreatAxios

@TheGreatAxios TheGreatAxios commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Closes CL-11528

Why
Nothing tested the live triage chain as it really runs: a webhook reaches the hub, the listener deploys a manager, and the manager deploys the judge and posts. The unit tests stub every hop, which is how a manager that crashed on any judge or code review reached main unnoticed (fixed in CL-11532).

What changed

  • New tests/chain/ suite, run with bun run test:chain and as its own CI step. It starts the real hub on a fresh Postgres database, with each sidecar as a local process. It then sets the workspace up through the hub's routes and publishes and upgrades the workflows with the portal's own Upgrade code. Finally it sends signed GitHub webhooks.
  • A fake GitHub stands in for the App, its repositories, their policy files and pull requests. It answers only the writes triage makes and records each one. Any other request is a miss, and every case checks that it had none. A stub decision model answers on System One's wire. No account, key or model spend is needed.
  • The suite has seven cases, each on its own repository.
  • The suite covers GitHub only, because there is no Bitbucket Data Center fake yet. Code review is not covered either: it needs an isolated sidecar, and a local process is not one.
  • The fake does not check auth headers, pages only a pull request's file list, and has no rate limits.
  • The CI job now has a 45-minute timeout.
  • One manager unit test is removed because the judge case now covers it: "rules that need the judge post nothing yet". The judge case checks for exactly one comment, one label and one recorded verdict (the judge's), which covers that test's check that done carries no report while the judge is out.
  • AGENTS.md (one line), docs/DEV.md and docs/IMPLEMENTATION.md describe the new suite.
Case On main
A policy of only deterministic rules posts its verdict at once and the manager stops passes
The hub records the verdict from the manager's done passes
A policy that needs the judge deploys it and posts the judge's verdict once, and only the judge's verdict is recorded passes
A head held for CI is marked waiting, and its check run's completion settles it passes (since CL-11533)
A push while the manager is up is mailed to it, and one after it stopped deploys the next, never two at once passes
A Do the policy does not run on its own is held, and its verdict records it as a held assign passes
A held Do approved through the hub runs once on the pull request passes

Unverified hypothesis
After the judge case, the pull request's claim was once left live, naming a manager run that had already stopped. The run logs suggest the judge's wake-up mail started a second body after done had ended the claim; that body took the claim again, and the run's own stop then aborted it before its done. The design treats this as harmless, since the next take finds the run gone. The test therefore checks that the manager's run is gone, not that the claim reads ended. I saw this once and have not reproduced it.

Verified

@linear-code

linear-code Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

CL-11528

CL-9988

@TheGreatAxios
TheGreatAxios force-pushed the cl-11528-add-an-end-to-end-test-of-the-live-triage-chain-from-webhook branch from 1d06dbc to 41b0093 Compare October 26, 2026 20:00
@TheGreatAxios
TheGreatAxios marked this pull request as ready for review October 26, 2026 20:00
@TheGreatAxios
TheGreatAxios merged commit 9aee453 into main Oct 26, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant