Repository navigation
fix(deploy): Allow the local-time pattern that Junior sends - #10
Merged
Merged
Conversation
The local-time pattern in roach.tfvars started with (?<![\w-]). Junior sends the pattern with NOT_AFTER_WORD. The service allows only equal patterns, so it refused every Junior run with HTTP 400. Co-Authored-By: David Cramer <david@sentry.io>
dcramer
approved these changes
Oct 15, 2026
2 tasks done
dcramer
added a commit
to getsentry/junior
that referenced
this pull request
Oct 16, 2026
Evals now use the shared Roach service at `https://roach-proxy.getsentry.net` ([getsentry/roach](https://github.com/getsentry/roach)), in place of a local proxy with recordings in git. This is a hard cutover. The service keeps the recordings of the tenant `getsentry/junior` in its GCS bucket, which deletes each one 30 days after it was written. **What changes** - `src/recording-run.ts` starts a run with `startRemoteRun()`. Workers get the run's `controlUrl` and token, and open one session per test as before. - `ROACH_TOKEN` selects the mode. With the token, the default is `auto`. Without it, the service allows only `replay`. `VITEST_EVALS_REPLAY_MODE` still overrides. Only global setup reads the token, and it removes it from `process.env`, so workers and child processes do not get it. - `src/roach/` is now a copy of the client side of getsentry/roach at `653c28e`: `client.ts`, `types.ts`, `report.ts`, and the patterns of `values.ts`. The local proxy, its CLI, and its tests are deleted. `@sentry/roach` is not on npm, and pnpm refuses it as a git dependency (`ERR_PNPM_GIT_DEP_PREPARE_NOT_ALLOWED`, because of its `prepare` script). `src/roach/README.md` tells how to update the copy. - `scripts/network-jail.sh` resolves `roach-proxy.getsentry.net` and allows it on TCP 443. The launcher for a local proxy outside the jail is gone. - The committed recordings in `packages/junior-evals/recordings/` are deleted. So are `evals-prune-recordings.yml` and the `collect-eval-recordings`, `commit-eval-recordings`, and `eval-recordings-mode` actions. The four eval workflows pass `ROACH_TOKEN` to the run step and no longer have a "commit recordings" job. - `AGENTS.md` and "Recordings" in `packages/junior-evals/README.md` describe the new flow. **Verification** - `pnpm --filter @sentry/junior-evals typecheck` passes. - On Node 24, I ran `startRecordingRun()` with the real rules of this branch against the deployed service, without a token. The service accepted the run. A `webFetch`-style GET and an AI Gateway POST came back `412 missed`. The report line and the non-zero exit code worked. Before that, the service refused Junior's `local-time` value pattern; getsentry/roach#10 fixed it. - The jail rules were checked with a stub `sudo`. They allow `136.69.81.163:443`, and the reject rules still come last. - I did not run the eval harness tests (`pnpm --filter @sentry/junior-evals test`) because the sandbox has no Postgres. The only harness test of Roach was the local proxy test, which this PR deletes. **Before merge** - Give `getsentry/junior` access to the `ROACH_TOKEN` organization secret ("Deploy", step 6, in the roach README). Without it, CI runs only replay, and every eval misses. - The bucket starts empty, so the first `auto` run of each suite calls the model live for every request. **Behavior changes to know** - A local run without `ROACH_TOKEN` uses `replay`. The service also refuses requests that no rule records, such as Vercel Sandbox API calls (HTTP 403). So behavioral and integration evals need the token to run locally. - Recordings are readable by anyone who can make the same request. The service redacts credential-like request headers, but Junior's own `SECRET_ENV` list for body redaction is gone, because the service has no per-run secrets. - Misses report only the key of the recording, not the closest recording and the differing parts. - `.gitattributes` and `warden.toml` still name `packages/junior-evals/recordings/**`. That keeps this large deletion out of review. They can be deleted in a follow-up. <!-- junior-request-attribution:start --> via Junior system actor `event`. <!-- junior-request-attribution:end --> <!-- junior-session-footer:start --> <!-- junior-conversation-id:slack%3AC0B595QDZLL%3A1791656969.630679 --> -- [View Junior Session](https://junior-prod.sentry.dev/conversations/slack%3AC0B595QDZLL%3A1791656969.630679) [[Sentry]](https://sentry.sentry.io/explore/conversations/slack%3AC0B595QDZLL%3A1791656969.630679/?project=4510944073809921) <!-- junior-session-footer:end --> --------- Co-authored-by: sentry-junior[bot] <264270552+sentry-junior[bot]@users.noreply.github.com> Co-authored-by: David Cramer <david@sentry.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The service refuses every Junior eval run with HTTP 400:
The
local-timepattern indeploy/gcp/roach.tfvarsstarted with(?<![\w-]). Junior sends it withNOT_AFTER_WORDfromvalues.ts(packages/junior-evals/src/recording-rules.ts). The service allows only patterns that are exactly equal. This change copies the pattern of Junior intovalue_patterns, and adds a comment that the two must stay equal. Theevent-idpattern was already equal.I checked this against the deployed service with
replayruns: the built-in patterns andevent-idare accepted (HTTP 201), and the currentlocal-timevalue is refused (HTTP 400).After merge: run
terraform apply -var-file=roach.tfvarsindeploy/gcp/, then restart the VM, because it reads its config only at start:The restart also fixes the
publicUrl: the running service still gives runshttps://roach-proxy.sentry.dev, because it did not restart after #6.Needed by the Junior change that moves evals to this service.
via David Cramer.
--
View Junior Session [Sentry]