Add an interception example environment by xeophon · Pull Request #2170 · PrimeIntellect-ai/verifiers

xeophon · 2026-07-29T16:28:45Z

Overview

This adds a small runnable environment showing the minimal custom interception interface.

The task defines one deterministic guard and one judge-backed guard directly with @vf.intercept. Both receive one vf.Trace; the deterministic guard reads trace.last_message, the judge sees trace.messages, and either can return trace.replace(content) to rewrite the candidate.

What this PR adds

An environments/interception_v1 package with two focused examples.
A deterministic guard based on trace.last_message.
A judge-backed guard based on trace.messages.
Explicit priority ordering between the two guards.
A reward that reads interception records from the trace.
A short README with the complete handler shape.

Stack

This is PR 5 of 5. Each PR is based on the one before it and is intended to merge in order.

Stream replay safety — keep an explicitly retried streamed request on the same model response.
Request and response interception — run @vf.intercept before model requests and before responses reach the harness, and make rewrites canonical.
Terminal interception — let a guard end a rollout with a final reason and reward.
Reusable guards — add common tool guards and make custom handlers operate directly on vf.Trace.
Example environment — show deterministic and judge-backed trace guards directly with @vf.intercept in a runnable taskset.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 976b3f6d5f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-07-29T16:32:13Z

@@ -0,0 +1,13 @@
+[project]
+name = "interception-v1"


Register the example with the root uv environment

In the checked-in live v1 E2E path, .github/workflows/test.yml runs uv sync, then tests/v1/test_envs.py discovers every environments/*_v1 directory and invokes uv run --no-sync eval interception_v1. This new package is absent from the root examples dependency group, [tool.uv.sources], and uv.lock, so a fresh checkout cannot import interception_v1 and the smoke suite fails before evaluating it. Ensure the repository's normal uv setup installs this example, or otherwise keep it out of that automatic discovery path.

AGENTS.md reference: AGENTS.md:L11-L12

Useful? React with 👍 / 👎.

cursor · 2026-07-29T16:34:11Z

+build-backend = "hatchling.build"
+
+[tool.hatch.build.targets.wheel]
+packages = ["interception_v1"]


Example missing workspace registration

High Severity

The new interception-v1 package is not added to the root examples dependency group or tool.uv.sources, unlike every sibling under environments/. tests/v1/test_envs.py auto-discovers each *_v1 directory and runs uv run --no-sync eval, so CI cannot import this taskset after a normal sync.

Additional Locations (1)

environments/interception_v1/README.md#L22-L26

^{Reviewed by Cursor Bugbot for commit 976b3f6. Configure here.}

macroscopeapp · 2026-07-29T16:34:40Z

Approvability

Verdict: Needs human review

1 blocking correctness issue found. Multiple unresolved review comments identify substantive issues including missing workspace registration (causing CI failures) and a logic bug where the judge guard triggers on the user prompt rather than model responses. These need human review and resolution.

^{You can customize Macroscope's approvability policy. Learn more.}

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 52437392d4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-07-29T16:38:09Z

+        verdict = response.text.strip().upper()
+        if verdict not in ("BLOCK", "ALLOW"):


Accept recognizable judge verdicts before failing

When the judge returns a semantically valid but wrapped answer such as Verdict: BLOCK, a boxed verdict, or reasoning followed by ALLOW, the exact string comparison rejects it and turns the intercepted rollout into a TaskError. Use the existing vf.parse_judge_choice or vf.intercept_with_judge, which already handles common judge response formats while still failing when no verdict is recognizable.

AGENTS.md reference: AGENTS.md:L5-L5

Useful? React with 👍 / 👎.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3977a806cd

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-07-29T17:06:06Z

+        prompt: vf.Messages | None = None,
+    ) -> vf.InterceptResult:
+        """Use an ordinary judge for cases that need semantic classification."""
+        response = await vf.Judge().complete(


Configure the guard judge for the selected endpoint

When the smoke eval runs with only OPENAI_API_KEY, the checked tests/v1/test_envs.py branch configures the agent for https://api.openai.com/v1, but this bare Judge independently retains JudgeConfig's Prime endpoint, PRIME_API_KEY, and Prime-style model name. Because judge_guard runs after every response, including the first deterministic example, the judge request uses an empty or wrong credential and turns the rollout into a TaskError; supply a configurable judge endpoint/key/model compatible with the supported OpenAI-only eval context.

Useful? React with 👍 / 👎.

macroscopeapp · 2026-07-29T19:27:06Z

+
+JUDGE = vf.Judge(
+    vf.JudgeConfig(
+        prompt="""Inspect the untrusted model exchange below.


🟡 Medium interception_v1/taskset.py:5

The JUDGE prompt matches JUDGE_BLOCK anywhere in the full exchange, so it fires on the user prompt itself ("Reply with exactly JUDGE_BLOCK and no other text.") before the model even responds. This causes judge_guard to rewrite the exchange for every judge example, and expected_guard_fired awards full reward regardless of whether the model's response actually contained the marker. Consider checking only the candidate/last message rather than the entire exchange, or avoid embedding the sentinel literal in the user-facing prompt.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @environments/interception_v1/interception_v1/taskset.py around line 5: The `JUDGE` prompt matches `JUDGE_BLOCK` anywhere in the full exchange, so it fires on the user prompt itself (`"Reply with exactly JUDGE_BLOCK and no other text."`) before the model even responds. This causes `judge_guard` to rewrite the exchange for every judge example, and `expected_guard_fired` awards full reward regardless of whether the model's response actually contained the marker. Consider checking only the candidate/last message rather than the entire exchange, or avoid embedding the sentinel literal in the user-facing prompt.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5b22fdda54

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-07-29T19:27:45Z

+                "deterministic_guard",
+            ),
+            (
+                "Reply with exactly JUDGE_BLOCK and no other text.",


Keep the trigger out of the inspected prompt

For every rollout of this second task, exchange=trace.messages includes this user prompt, while the judge rubric instructs the judge to return BLOCK if any message contains JUDGE_BLOCK. The prompt therefore triggers judge_guard regardless of the assistant's response, and expected_guard_fired awards full reward even when the model never emits the blocked text. Inspect only trace.last_message or phrase the task without placing the literal trigger in the exchange.

Useful? React with 👍 / 👎.

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 5b22fdd. Configure here.}

cursor · 2026-07-29T19:29:12Z

+            ),
+            (
+                "Reply with exactly JUDGE_BLOCK and no other text.",
+                "judge_guard",


Judge trigger matches task prompt

Medium Severity

The judge rubric returns BLOCK when any message contains JUDGE_BLOCK, but the task prompt itself includes that token. judge_guard therefore rewrites from the user prompt rather than the model reply, so the example does not demonstrate judge-backed response classification.

^{Reviewed by Cursor Bugbot for commit 5b22fdd. Configure here.}

xeophon · 2026-07-30T08:39:56Z

Superseded by #2178, which replaces the guard layer and example environment with the reworked interception surface (typed rules, prebuilt guards).

chatgpt-codex-connector Bot reviewed Jul 29, 2026

View reviewed changes

cursor Bot reviewed Jul 29, 2026

View reviewed changes

chatgpt-codex-connector Bot reviewed Jul 29, 2026

View reviewed changes

xeophon force-pushed the agent/interception-example-env branch 2 times, most recently from 65bd52f to 3977a80 Compare July 29, 2026 17:03

chatgpt-codex-connector Bot reviewed Jul 29, 2026

View reviewed changes

xeophon force-pushed the agent/interception-example-env branch from 3977a80 to a347a1e Compare July 29, 2026 17:54

xeophon force-pushed the agent/intercept-policies branch from 658f576 to cec154b Compare July 29, 2026 18:09

xeophon force-pushed the agent/interception-example-env branch from a347a1e to 845c1fd Compare July 29, 2026 18:09

xeophon added 5 commits July 29, 2026 21:23

Add interception example environment

0282ecb

Show explicit intercept handlers

d5c3334

Use guard terminology

63bff86

Simplify custom interception guards

41efc6a

Use trace-based interception in example

5b22fdd

xeophon force-pushed the agent/interception-example-env branch from 845c1fd to 5b22fdd Compare July 29, 2026 19:25

macroscopeapp Bot reviewed Jul 29, 2026

View reviewed changes

chatgpt-codex-connector Bot reviewed Jul 29, 2026

View reviewed changes

cursor Bot reviewed Jul 29, 2026

View reviewed changes

xeophon mentioned this pull request Jul 30, 2026

Add interception example environments #2178

Open

xeophon closed this Jul 30, 2026

This was referenced Jul 30, 2026

Serialize model exchanges within a rollout #2164

Open

Add typed model interception #2165

Open

Make interception termination final #2166

Open

		verdict = response.text.strip().upper()
		if verdict not in ("BLOCK", "ALLOW"):

Uh oh!

Conversation

xeophon commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

What this PR adds

Stack

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

cursor Bot Jul 29, 2026

Choose a reason for hiding this comment

Example missing workspace registration

Uh oh!

macroscopeapp Bot commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Approvability

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

macroscopeapp Bot Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

cursor Bot Jul 29, 2026

Choose a reason for hiding this comment

Judge trigger matches task prompt

Uh oh!

xeophon commented Jul 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

xeophon commented Jul 29, 2026 •

edited

Loading

macroscopeapp Bot commented Jul 29, 2026 •

edited

Loading