Repository navigation
feat(evals): Record and replay evals with the Roach service - #2156
Merged
Merged
Conversation
Evals now start a run on the shared Roach service at roach-proxy.getsentry.net, in place of a local proxy with recordings in git. The service keeps the recordings in its GCS bucket. - Copy the client of getsentry/roach into src/roach/, and delete the local proxy, its CLI, and its tests. - ROACH_TOKEN selects the mode: auto with the token, replay without it. Only global setup reads the token. - The network jail allows the Roach service on HTTPS. - Delete the committed recordings and the CI that committed, chose the mode for, and pruned them. Eval workflows pass ROACH_TOKEN instead.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Sentry Evalsjunior-behavioral
Base commit junior-guardian
Base commit junior-integration
Base commit junior-router
Base commit |
MSW blocked the session calls that each eval test makes to roach-proxy.getsentry.net, so every integration eval failed before it ran. The local proxy was on 127.0.0.1, which MSW already let through.
The local proxy redacted the values of AI_GATEWAY_API_KEY, SENTRY_EVALS_API_KEY, VERCEL_OIDC_TOKEN, and VERCEL_TOKEN in every recording. The service only knew credential headers, so a response that showed one of these values was written to the public bucket. The run config now sends them as secrets (getsentry/roach#11). A service without that change ignores the field. Co-Authored-By: David Cramer <david@sentry.io>
A new origin or value pattern now needs only a change to recording-rules.ts. getsentry/roach#12 removes the copies in roach.tfvars. Co-Authored-By: David Cramer <david@sentry.io>
Co-Authored-By: David Cramer <david@sentry.io>
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Evals now use the shared Roach service at
https://roach-proxy.getsentry.net(getsentry/roach), in place of a local proxy with recordings in git. This is a hard cutover. The service keeps the recordings of the tenantgetsentry/juniorin its GCS bucket, which deletes each one 30 days after it was written.What changes
src/recording-run.tsstarts a run withstartRemoteRun(). Workers get the run'scontrolUrland token, and open one session per test as before.ROACH_TOKENselects the mode. With the token, the default isauto. Without it, the service allows onlyreplay.VITEST_EVALS_REPLAY_MODEstill overrides. Only global setup reads the token, and it removes it fromprocess.env, so workers and child processes do not get it.src/roach/is now a copy of the client side of getsentry/roach at653c28e:client.ts,types.ts,report.ts, and the patterns ofvalues.ts. The local proxy, its CLI, and its tests are deleted.@sentry/roachis not on npm, and pnpm refuses it as a git dependency (ERR_PNPM_GIT_DEP_PREPARE_NOT_ALLOWED, because of itspreparescript).src/roach/README.mdtells how to update the copy.scripts/network-jail.shresolvesroach-proxy.getsentry.netand allows it on TCP 443. The launcher for a local proxy outside the jail is gone.packages/junior-evals/recordings/are deleted. So areevals-prune-recordings.ymland thecollect-eval-recordings,commit-eval-recordings, andeval-recordings-modeactions. The four eval workflows passROACH_TOKENto the run step and no longer have a "commit recordings" job.AGENTS.mdand "Recordings" inpackages/junior-evals/README.mddescribe the new flow.Verification
pnpm --filter @sentry/junior-evals typecheckpasses.startRecordingRun()with the real rules of this branch against the deployed service, without a token. The service accepted the run. AwebFetch-style GET and an AI Gateway POST came back412 missed. The report line and the non-zero exit code worked. Before that, the service refused Junior'slocal-timevalue pattern; fix(deploy): Allow the local-time pattern that Junior sends roach#10 fixed it.sudo. They allow136.69.81.163:443, and the reject rules still come last.pnpm --filter @sentry/junior-evals test) because the sandbox has no Postgres. The only harness test of Roach was the local proxy test, which this PR deletes.Before merge
getsentry/junioraccess to theROACH_TOKENorganization secret ("Deploy", step 6, in the roach README). Without it, CI runs only replay, and every eval misses.autorun of each suite calls the model live for every request.Behavior changes to know
ROACH_TOKENusesreplay. The service also refuses requests that no rule records, such as Vercel Sandbox API calls (HTTP 403). So behavioral and integration evals need the token to run locally.SECRET_ENVlist for body redaction is gone, because the service has no per-run secrets..gitattributesandwarden.tomlstill namepackages/junior-evals/recordings/**. That keeps this large deletion out of review. They can be deleted in a follow-up.via Junior system actor
event.--
View Junior Session [Sentry]