Fix MCP session lifecycle and replay safety by xeophon · Pull Request #2134 · PrimeIntellect-ai/verifiers

xeophon · 2026-07-26T07:21:16Z

Overview

Centralize null and bash harness MCP handling into one lifecycle-aware client and make transport failures explicit, replay-safe, and diagnosable.

Details

own the MCP transport in a dedicated task so SDK cancellation scopes stay with their owner
preserve outer cancellation while bounding cancellation-time session teardown
distinguish replay-safe connection failures from calls whose delivery is unknown
prevent automatic replay when a non-idempotent tool call may have reached the server
emit structured diagnostics for the parent harness and retry policy
remove the duplicated null-harness MCP client by sharing the null and bash program

Note

Medium Risk
MCP delivery-unknown handling and restricted-runtime reuse change failure and isolation behavior on eval retries and multi-agent Docker; NeMo Gym adds a large optional dependency tree (Ray, wandb) for Python 3.12+.

Overview
Adds NeMo Gym v1 support: a NeMoGymTaskset that seeds/verifies against Gym resource servers while verifiers harnesses run the agent loop, optional NeMoGymEnv to start packaged servers, a verifiers[nemo-gym] extra, docs, and the nemo-gym-weather-v1 example.

MCP harness programs are merged into one shared minimal/program.py (bash gated by --bash). Transport runs in a dedicated task; failures emit VF_MCP_ERROR= JSON that the parent maps to MCPTransportError or MCPDeliveryUnknownError. Retries skip replay when delivery is unknown unless RetryConfig explicitly allows MCPDeliveryUnknownError.

Restricted runtimes can be reused across rollouts: runtime.rollout() replaces prepare_setup, kills stray processes, resets caches, and toggles network policy via _apply_network_policy. Rollouts enter that context; agentic-judge shared Docker uses a blocklist in e2e.

Also exports mcp_session on the launch path, tightens builtin_harness_ids to packages with a harness module, and propagates network_allow/network_block into judge tasks.

^{Reviewed by Cursor Bugbot for commit 53b52ed. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Fix MCP session lifecycle and replay safety in harness program

Replaces separate bash and null harness programs with a shared minimal program that manages MCP HTTP transports with retry logic, emitting structured VF_MCP_ERROR= markers for the parent harness to parse.
Adds two new error types — MCPTransportError and MCPDeliveryUnknownError — so callers can distinguish clean transport failures from cases where a non-replayable mutation may have already been accepted by the server.
Updates automatic retry logic in retries.py to skip replays when MCPDeliveryUnknownError is present, unless explicitly opted into via retry config.
Adds execution_prepared tracking and a rollout() context manager to the runtime base class, enabling Docker and Prime runtimes to toggle between unrestricted trusted-setup policy and restricted execution network policy per rollout.
Adds a NeMo Gym taskset and environment that manage the lifecycle of a NeMo Gym MCP resources server per worker, with MCP session seeding, tool discovery, and reward evaluation via the server.
Risk: automatic retries for MCP-based tools now require explicit opt-in when any delivery uncertainty exists, which may change retry behavior for existing MCP-backed tasksets.

^{Macroscope summarized 53b52ed.}

macroscopeapp · 2026-07-26T07:27:43Z

Approvability

Verdict: Unable to determine

Macroscope's correctness review was unable to post its findings for this PR. Approvability cannot proceed without a successful correctness review.

^{You can customize Macroscope's approvability policy. Learn more.}

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 67e0acc36c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

mikasenghaas

do we have some data/ experiments to backup the robustness improvs around mcp?

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9789a1d4b4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: fc39d94a2c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

cursor

Cursor Bugbot has reviewed your changes and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit d9c602c. Configure here.}

mikasenghaas

not super convinced

mikasenghaas · 2026-07-29T17:09:48Z

+class MCPTransportError(HarnessError):
+    """A harness-owned MCP session lost its transport; the message is structured JSON."""
+
+
+class MCPDeliveryUnknownError(MCPTransportError):
+    """An MCP transport failed after a non-replayable operation may have been accepted."""


lets not try to reinvent finegrained error cls and lets just have the error that is raised in the harness bubble up. we just wrap with harness error to have a rough idea of the "location" of the error

mikasenghaas · 2026-07-29T17:10:02Z

+                if (
+                    isinstance(payload, dict)
+                    and payload.get("delivery")
+                    in {"operation_not_started", "response_unknown"}
+                    and isinstance(payload.get("replay_safe"), bool)
+                ):
+                    error_cls = (
+                        MCPDeliveryUnknownError
+                        if payload["delivery"] == "response_unknown"
+                        and not payload["replay_safe"]
+                        else MCPTransportError
+                    )
+                    mcp_error = error_cls(diagnostic)
+            if isinstance(mcp_error, MCPDeliveryUnknownError):
+                raise mcp_error


then we can also remove those shenannigans

mikasenghaas · 2026-07-29T17:11:17Z

+        m.name
+        for m in pkgutil.iter_modules(harnesses.__path__)
+        if m.ispkg
+        and importlib.util.find_spec(f"{harnesses.__name__}.{m.name}.harness")


bit ugly to break with our one folder = one harness convention but ig this is the lesser evil than duplicated code?

mikasenghaas · 2026-07-29T17:12:50Z

-    its trace is retryable (all captures count, not just the most recent)."""
+    its trace is retryable (all captures count, not just the most recent).
+    A possibly-delivered MCP mutation vetoes replay unless explicitly included."""
+    replay_unknown = (


hmmm im not sure im loving this. in between who and who is this error raised? if its between harness + tool server we should fix in the harness and not patch here. retrying is really last resort in my mind

chatgpt-codex-connector · 2026-07-31T21:17:49Z

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

macroscopeapp Bot reviewed Jul 26, 2026

View reviewed changes

Comment thread verifiers/v1/retries.py Outdated

cursor Bot reviewed Jul 26, 2026

View reviewed changes

Comment thread verifiers/v1/harnesses/mcp_client.py Outdated

chatgpt-codex-connector Bot reviewed Jul 26, 2026

View reviewed changes

Comment thread verifiers/v1/retries.py Outdated

This was referenced Jul 26, 2026

Preserve MCP tool error outcomes #2135

Closed

Preserve MCP tool error outcomes #2138

Open

xeophon force-pushed the agent/fix-mcp-session-lifecycle branch from fffc408 to 18109b8 Compare July 27, 2026 13:28

xeophon requested a review from mikasenghaas July 27, 2026 13:30

cursor Bot reviewed Jul 27, 2026

View reviewed changes

Comment thread verifiers/v1/harnesses/mcp_client.py Outdated

mikasenghaas reviewed Jul 28, 2026

View reviewed changes

Comment thread verifiers/v1/harnesses/bash/program.py Outdated

Comment thread verifiers/v1/harnesses/minimal/program.py

Comment thread verifiers/v1/harnesses/mcp_client.py Outdated

xeophon mentioned this pull request Jul 28, 2026

Retry replay-safe MCP tool calls #2154

Open

chatgpt-codex-connector Bot reviewed Jul 28, 2026

View reviewed changes

Comment thread verifiers/v1/harness.py

cursor Bot reviewed Jul 28, 2026

View reviewed changes

Comment thread verifiers/v1/harnesses/minimal/program.py

macroscopeapp Bot reviewed Jul 28, 2026

View reviewed changes

Comment thread verifiers/v1/harness.py

chatgpt-codex-connector Bot reviewed Jul 28, 2026

View reviewed changes

Comment thread verifiers/v1/harnesses/minimal/program.py

cursor Bot reviewed Jul 28, 2026

View reviewed changes

Comment thread verifiers/v1/harnesses/minimal/program.py

xeophon force-pushed the agent/fix-mcp-session-lifecycle branch from fc39d94 to d9c602c Compare July 29, 2026 09:18

cursor Bot reviewed Jul 29, 2026

View reviewed changes

Comment thread verifiers/v1/harness.py Outdated

Comment thread verifiers/v1/harnesses/minimal/program.py

xeophon requested a review from mikasenghaas July 29, 2026 14:44

mikasenghaas reviewed Jul 29, 2026

View reviewed changes

xeophon added 10 commits July 31, 2026 21:53

Add NeMo Gym V1 integration

926c067

Address NeMo Gym review feedback

0a3d1ed

Simplify NeMo Gym V1 adapter

d70357d

Further simplify NeMo Gym adapter

9fa3dde

Adapt NeMo Gym taskset to current v1 APIs

774f75a

Reuse NeMo Gym resource servers directly

ddea111

Add NeMo Gym optional dependency

1243d82

Trim NeMo Gym extra dependencies

bc37a93

Reuse shared MCP sessions for NeMo Gym

5a013cd

Simplify NeMo Gym adapter internals

c8e9a26

xeophon added 26 commits July 31, 2026 21:53

Normalize NeMo Gym tool outputs

ff506ce

Adapt NeMo Gym integration to Ruff 0.16

5c2c113

Eliminate NeMo Gym port selection race

5e4d470

Refresh NeMo Gym lock markers

523ab69

Adapt NeMo Gym to consolidated decorators

44df475

Support restricted shared agent runtimes

172c2e8

Simplify shared runtime lifecycle

eceaab7

Reuse restricted runtime policy lifecycle

0749484

Preserve peer setup lifecycle

9f7e1db

Prepare reused runtime routes before tools

70bf919

Reopen trusted setup between shared rollouts

5367746

Kill prior processes before setup reopen

57bc465

Close reused runtime process fork race

314268a

Reset trusted setup caches between agents

18d1e91

Fail closed on unkillable agent processes

f94606c

Isolate restricted runtime reuse from agents

a52475f

Harden restricted runtime reuse

d89955a

Run reused agents as the image user

de6f131

Fix MCP session lifecycle

23b4734

Fix MCP retry and teardown edge cases

b1bbad7

Preserve grouped MCP cancellation

2234094

Use one shared null and bash program

462b219

Move shared harness program under minimal

5faff0f

Preserve MCP failure classification

8252ad2

Preserve MCP replay veto on sandbox death

ea596b1

Fix decorator import after restack

53b52ed

xeophon force-pushed the agent/fix-mcp-session-lifecycle branch from b20f3fb to 53b52ed Compare July 31, 2026 21:17

xeophon changed the base branch from main to agent/fix-agentic-judge-restricted-runtime July 31, 2026 21:18

Uh oh!

Conversation

xeophon commented Jul 26, 2026 • edited by macroscopeapp Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Details

Fix MCP session lifecycle and replay safety in harness program

Uh oh!

Uh oh!

Uh oh!

macroscopeapp Bot commented Jul 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Approvability

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

mikasenghaas left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

mikasenghaas left a comment

Choose a reason for hiding this comment

Uh oh!

mikasenghaas Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

mikasenghaas Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

mikasenghaas Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

mikasenghaas Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot commented Jul 31, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

xeophon commented Jul 26, 2026 •

edited by macroscopeapp Bot

Loading

macroscopeapp Bot commented Jul 26, 2026 •

edited

Loading