feat: self-healing prime tunnels and typed tunnel failures

[mikasenghaas]

Summary

• Tunnel.expose() now yields an Endpoint handle (url() / healthy(url)) instead of a URL string — a tunnel's URL is not a constant, so the interception server asks per acquire instead of caching a start-time snapshot.
• PrimeTunnel's endpoint self-heals: url() verifies the frpc process on every call and the server-side registration behind a 60s TTL, and re-mints a dead tunnel (through the existing retry + rate-limiter path) at a new public URL before the next rollout gets its slot. The probe reads get_tunnel().status rather than check_registered() — deletion is soft server-side (status="terminated", GET keeps returning 200), so existence alone reads a terminated tunnel as registered.
• Mid-rollout tunnel death is now typed: on a harness failure the rollout probes the interception server's endpoint, and if the tunnel was down, fails with TunnelError — the HarnessError (with the gateway's 404 body) chained as cause. Infra drops become retryable and distinguishable from agent failures instead of counting as HarnessError: bash exited 1: <404 HTML>.
• The probe is anchored to the failing rollout's own slot URL (healthy(url)): a re-mint always changes the URL, so a stale slot URL proves that rollout's tunnel died even when a concurrent acquire has already healed past it — a bare liveness probe would race the heal and misattribute the failure.
• CustomTunnel yields a FixedEndpoint (always healthy — the operator owns that endpoint); no behavior change.

Breaking

• Tunnel.expose() yields an Endpoint instead of str — custom Tunnel implementations must yield an object with async url() -> str / async healthy(url: str) -> bool (or a FixedEndpoint).
• InterceptionServer.base_url is removed; use await server.url().

Verification

Live evals (echo-agentic-v1, bash harness, prime runtime, -c 1 so rollouts run back-to-back on one shared elastic-pool tunnel), interrupting at a rollout's first intercepted model turn:

• frpc killed mid-rollout 2 — before: every later rollout on the server would keep failing at its first model call as HarnessError with the 404 HTML body. After: rollout 2 fails typed (stop=TunnelError, trace records type: TunnelError with the 404-HTML HarnessError as chained cause), the next acquire re-mints a new tunnel on the same local port, and the remaining rollouts all score reward=1.000. Re-run after the anchored-probe fix with the same result.
• registration terminated server-side mid-rollout 2 of 4 — the gateway routes by the live frpc session, so the data plane survives the soft delete; at the first acquire past the TTL: registration terminated server-side → re-mint → reward=1.000.
• The heal-vs-probe race (tunnel dies, a concurrent acquire re-mints, the failed rollout probes last) is covered deterministically with a mocked prime_tunnel SDK: the stale slot URL reads dead while the replacement reads healthy.

uv run pytest tests/v1 (64 passed, 63 key-gated skips), ruff check, and pre-commit green on touched files.

🤖 Generated with Claude Code

Note

Add self-healing prime tunnels and typed TunnelError failures to rollout

• Introduces an Endpoint protocol in base.py with async url() and healthy() methods, replacing raw URL strings yielded by Tunnel.expose().
• Adds PrimeEndpoint in prime.py that auto-heals dead tunnels by reminting a new frpc connection when the existing one is unhealthy, with rate limiting and registration health checks.
• InterceptionServer now stores an Endpoint abstraction instead of a static base_url, and resolves the public URL dynamically at acquisition time.
• RolloutRun in rollout.py detects tunnel outages during harness failures via the new _tunnel_failure() method and surfaces them as TunnelError instead of generic HarnessError.
• RolloutSession gains a server field to hold a back-reference to its InterceptionServer, enabling post-execution health checks.

^{Macroscope summarized 34b1b64.}

---

Note

Medium Risk
Changes remote-eval networking (tunnel lifecycle, error classification on harness failures) and breaks custom Tunnel / callers that cached base_url, though healing and attribution are scoped to prime tunnels and explicit health probes.

Overview
Self-healing prime tunnels and typed infra failures replace cached tunnel URLs with per-acquire Endpoint handles (url() / healthy(url)). PrimeEndpoint re-mints when frpc dies or server-side registration is terminated (status probe, not bare existence), so later rollouts on a shared elastic pool get a fresh public URL without manual restart.

Interception and rollouts resolve await server.url() on each slot acquire instead of caching base_url at server start. When a harness exits with HarnessError and no stashed model error, the rollout probes healthy() against that rollout's slot URL (stale URL ⇒ dead tunnel even if another acquire already healed), and records TunnelError with the harness error chained—so 404 HTML from a gone tunnel is retryable infra, not agent failure.

Breaking: Tunnel.expose() yields Endpoint (custom tunnels need url() / healthy() or FixedEndpoint); InterceptionServer.base_url is removed in favor of await server.url().

^{Reviewed by Cursor Bugbot for commit 34b1b64. Bugbot is set up for automated code reviews on this repo. Configure here.}

[macroscopeapp]

Approvability

Verdict: Needs human review

1 blocking correctness issue found. This PR introduces significant new runtime behavior (self-healing tunnels with automatic re-minting and health probing) and has two unresolved review comments identifying a race condition in the close() method that could leak tunnel processes.

^{You can customize Macroscope's approvability policy. Learn more.}

[macroscopeapp]

🟡 Medium tunnel/prime.py:136

close() does not take self._lock, so a concurrent url() call can mint a replacement tunnel after teardown has already stopped the old one. If url() is awaiting _alive() when close() runs, close() sets _client = None and stops the old client; then url() sees _client is None and calls _mint(), creating a new frpc process and public URL. Because close() only stopped the old client, this replacement tunnel is leaked beyond the context lifetime. Acquire self._lock in close() so it cannot run concurrently with url() or healthy().

Suggested change

	async def close(self) -> None:
	"""Stop the *current* frpc client (it changes across heals). Runs the synchronous
	stop to completion even under cancellation (`run_shielded` re-raises the
	cancellation after); tunnel-stop failures are best-effort."""
	client, self._client = self._client, None
	if client is not None:
	with contextlib.suppress(Exception):
	await run_shielded(asyncio.to_thread(client.sync_stop))
	async def close(self) -> None:
	"""Stop the *current* frpc client (it changes across heals). Runs the synchronous
	stop to completion even under cancellation (`run_shielded` re-raises the
	cancellation after); tunnel-stop failures are best-effort."""
	async with self._lock:
	client, self._client = self._client, None
	if client is not None:
	with contextlib.suppress(Exception):
	await run_shielded(asyncio.to_thread(client.sync_stop))

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @verifiers/v1/interception/tunnel/prime.py around lines 136-143:

`close()` does not take `self._lock`, so a concurrent `url()` call can mint a replacement tunnel *after* teardown has already stopped the old one. If `url()` is awaiting `_alive()` when `close()` runs, `close()` sets `_client = None` and stops the old client; then `url()` sees `_client is None` and calls `_mint()`, creating a new frpc process and public URL. Because `close()` only stopped the *old* client, this replacement tunnel is leaked beyond the context lifetime. Acquire `self._lock` in `close()` so it cannot run concurrently with `url()` or `healthy()`.

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 6fefddb. Configure here.}

[cursor]

Endpoint close races without lock

Medium Severity

expose's finally calls PrimeEndpoint.close() without taking _lock, while url and healthy mutate _client under that lock across awaits. Teardown can interleave with an in-flight probe or mint, orphaning a started tunnel or stopping a client another coroutine still uses.

Additional Locations (1)

• verifiers/v1/interception/tunnel/prime.py#L53-L75

 

^{Reviewed by Cursor Bugbot for commit 6fefddb. Configure here.}