Skip to content

Implement ai.external: ExternalHarness, ExternalHarnessProvider and Session - #346

Merged
anbuzin merged 26 commits into
mainfrom
workers-claude-code-pi
Nov 18, 2026
Merged

anbuzin merged 26 commits into
mainfrom
workers-claude-code-pi

Conversation

@anbuzin

@anbuzin anbuzin commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

A draft for ai.external. #347, the competing draft, is closed.

An external harness is a coding-agent CLI (Codex, Claude Code, Pi, fx) where a model goes. It has the shape of an ai.Model: plain data and a provider. The state is in a session.

  • ExternalHarness: the settings of the CLI. It holds no state.
  • ExternalHarnessProvider: name, executable, env, and the stream() that ai.stream and ai.Agent.run call.
  • Session: the state (id, messages) and direct control (start, send, events, interrupt, close).
harness = ai.external.Codex(cwd=".")
async with harness.session() as session:
    async with agent.run(harness, messages) as run: ...
    await session.send(ai.user_message("And the tests?"))

Sessions

  • async with harness.session() is the only way to start the CLI. A model call outside the block raises ExternalHarnessError. No call starts a CLI for one turn.
  • harness.session() is a new session. harness.session(session_id) continues one that the CLI has, and harness.session(messages=history) starts one with a past (lossy). To continue later, save session.id with the harness data.
  • A harness has one open session in a task: a second harness.session() in the block raises. Two tasks can each open one.
  • session.start() starts the CLI early in an open block. It does not open a session.

Compared with #347

  • Core changes less. ai.stream still calls model.provider.stream(model, ...); the diff outside ai/external is Model -> ModelLike annotations, one aclosing block in api.py, the Context.model annotation, and the two protocols (ModelLike, ProviderLike) in models/core/model.py. ai/__init__.py imports external, as it does durable and testing. hooks.py is as on main.
  • The state and direct control are on a separate Session, not on the harness.
  • Not here, and in Implement ai.workers: one object per CLI session #347: approvals through hooks, output_type, fork() and sync().

fx

ai.external.Fx drives fx acp (Agent Client Protocol over stdio). The wire follows Codex; JsonRpcPeer moved to external/_jsonrpc.py so that both use it.

  • Fields: model_provider and effort. fx runs with --permission-mode full-access.
  • A new session with a past is written into the fx session store, with tool calls, results and images, and opened with session/load. If fx refuses it, the wire removes what it wrote and raises ExternalHarnessError with the fx version.
  • Limits, stated on Fx: a bridge tool must answer in 5 minutes; streamed tool results are short previews; instructions are fixed when the session starts; the built-in tools cannot be limited. The store format is internal to fx, so a new fx release can break the seed.
  • Session.interrupt() does nothing between the end of a turn and the start of the turn that fx makes from a late steer. After an interrupt, a steer that fx dropped goes again as a new prompt.

Status

  • Offline tests, ruff, mypy (src tests) and ty pass locally and in CI (Python 3.12 to 3.14) on 0073da8.
  • The live tests (AI_EXTERNAL_LIVE=1 uv run pytest tests/external/test_live.py) were not run after the session changes (d0dffe8 and later). The last recorded run is fx only, with fx 0.0.17 on 57bd861.

A worker is one session of a coding-agent CLI. It obeys the model
protocol (ai.stream, ai.Agent.run) and has a small direct API: start,
send, events, interrupt, close. Its state is plain data: settings,
session_id and portable ai messages. See DESIGN.md.

- L0: a private wire per CLI (_wire.Wire) that turns native records
  into ai events, Accepted and TurnEnd items.
- L1: Worker; one pump task owns the conversation; many readers;
  steers split the reply at a clean point; ai tool calls through the
  MCP bridge pause the reply like a model's tool call; telemetry spans
  for the session, each message and each CLI tool call.
- L2: Worker.stream(), built once on L1; cancel or stop-reading
  interrupts the turn.
- Codex over app-server.
- ClaudeCode, through claude-agent-sdk (new extra `claude-code`).
  Acceptance is the CLI's --replay-user-messages echo, matched by uuid.
- Pi, through `pi --mode rpc`, with a small extension that registers
  the MCP bridge. Acceptance is pi's user message_start.
- Process and spawn move from codex/jsonrpc.py to _process.py, now that
  Codex and Pi share them.
- TurnEnd.metadata goes on the turn's last reply, so a seeded pi
  continues on the model that wrote it.
- Live tests and examples cover all three CLIs.
- Split each provider into worker.py (settings) and wire.py (CLI protocol).
- Keep shared wire helpers in _process (spawn, probe) and _convert (files,
  builtin events, handoff); remove _handoff. _wire holds only the contract.
- Wires pass their own accepted file types and auth phrases.
- The bridge turns tool results into MCP text.
- Inline history reconciliation into Worker.stream(); label the session
  state that only stream() reads.
- Core: Context no longer knows about workers; data loads as ai.Model.
  Worker-specific wording removed from core comments.
- Rename provider tests to test_wire.py.
A worker now has the shape of an ai.Model: plain data and a provider.

- Core: ai.stream dispatches through model.provider.stream() again, with
  no branch for non-models. ModelLike and ProviderLike are protocols in
  models/core/model_like.py; model.py and ai/__init__.py are as on main.
  Context.model takes a model-like through a pydantic annotation; the
  custom validator and serializer are gone.
- WorkerProvider (CodexProvider, ClaudeCodeProvider, PiProvider) holds
  name, executable and env, makes the wire, and implements stream() for
  the model protocol.
- Session holds direct control: start, send, events, interrupt, close.
  `async with worker.session()` scopes the CLI, and a model call needs an
  open session. The CLI starts on first use, when the tools and the past
  are known, so it is not started twice. Readers stay subscribed when new
  tools restart the CLI.
- send() takes a Message only.
- Pi.provider (pi's --provider) is now Pi.model_provider.
- Live tests are opt-in through AI_WORKERS_LIVE=1, not a global pytest
  marker filter. ai no longer imports ai.workers at the top level.
@vercel

vercel Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
ai-python Ready Ready Preview, v0 Nov 17, 2026 11:45pm UTC

@anbuzin anbuzin added the feature New functionality label Nov 10, 2026
This was referenced Nov 10, 2026
@socket-security

socket-security Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff Package Supply Chain
Security
Vulnerability Quality Maintenance License
Addedpypi/​claude-agent-sdk@​0.2.17999100100100100

View full report

@vercel-internal-security-reviewer

Copy link
Copy Markdown

Security review details

A worker now goes where a model goes with no ceremony. The conversation
state is in the session; the worker keeps only the id of its session.

- Lifetime: without an open session, a model call starts the CLI,
  continues worker.session_id, and stops the CLI when the turn ends. The
  CLI stays alive through a tool loop. In `async with worker.session()`
  the same calls run on one live CLI.
- Worker.session_id is a setting. The worker's first new session sets
  it, so a saved worker continues the same CLI session later.
  Worker.messages is gone: the conversation is session.messages.
- worker.session() gives the worker's session, or a new one.
  worker.session(session_id) continues another session, and
  worker.session(messages=history) makes one that starts with a past.
- A context variable holds the open sessions, so a model call finds the
  live session of its worker, and two tasks can each open a session on
  one worker object.
- start() and close() are a pair and work without a block.
- events() reads from where the last reader stopped: send() and then
  events() loses nothing, and a reader that stops needs no aclose(). One
  reader. Events that nobody reads during a whole turn are dropped.
- workers.is_interrupted is gone. provider_metadata["worker"] marks a
  message that the worker wrote outside a complete model call:
  {"stop_reason": "interrupted"}, {"error": text} or {"synthetic": True}.
  The history comparison ignores all three, so a call after a failed
  turn works as a call after an interrupt does.
- A history with nothing new raises TranscriptMismatchError, not
  ValueError.
- Wire.start() takes session_id as an argument.
Nine examples in examples/workers, each the same task as the script of
the same name in examples/harnesses of #318, on the worker API.

- basic, streaming, steering, resume, combine, until_done, as_tool and
  custom_loop. handoff has the Claude Code to Codex steps only; its
  sandbox step needs a workspace layer that workers do not have.
- streaming.py is now the port. It keeps the call that sends the history
  back; its save-and-load part is in resume.py. It no longer takes the
  CLI as an argument.
- The files have the form of examples/agents: a one-line docstring,
  main() and the __main__ guard, with a fixed CLI.
- examples/workers/__init__.py: without it mypy stops on a duplicate
  module name (basic.py and custom_loop.py are also in examples/agents).
- run-examples.py has a --workers group with all worker examples; --all
  includes it. They need the claude and codex CLIs, logged in.
- examples/README.md lists workers/.
…el.py

No behavior change, but for `import ai`, which now gives `ai.workers`.

- ai/__init__.py imports `workers`, as it does `durable` and `testing`.
  The examples and the docstrings use `ai.workers.X`, with no
  `from ai import workers`.
- ModelLike and ProviderLike are in models/core/model.py, adjacent to
  Model. models/core/model_like.py is gone: the framework keeps a
  protocol in the module that owns the concept. The tests are in
  tests/models/core/test_model.py.
- workers/_convert.py is now workers/_utils.py. Its functions say what
  they do: extract_data, format_text_file, raise_for_unsupported_parts,
  get_builtin_call_events, get_builtin_result_event and
  count_reasoning_parts. The parameters are cli_name,
  accepted_media_types, accepted_description and started_call_ids.
- result_text is gone: its one line is inline in the three handoff
  modules, as it was in _bridge.py.
- workers/_session.py: private names start with a verb. _Run is _CliRun,
  with run_in_pump, send_tool_results and _handle_* for the wire items.
  Session has _mark_open, _stop_cli, _publish_event, _pending_events and
  _stream_queues. The public API and _sync keep their names.
- The worker examples read events with `match`, as the model examples
  do.
- The durable examples keep the `isinstance(context.model, ai.Model)`
  assertion, with a TODO: it can go if Context and Agent become generic
  on the model type, but then worker users must give the parameter.
No behavior change.

JsonRpcPeer has nothing specific to Codex, and the fx worker speaks
JSON-RPC over stdio too. It moves out of codex/ as Process and spawn
did when Pi started to share them. Its tests move with it, to
tests/workers/test_jsonrpc.py.
`ai.workers.Fx` drives `fx acp` (Agent Client Protocol over stdio), with
Codex as the model for the wire. `Fx` adds two fields, `model_provider`
and `effort`. fx runs with `--permission-mode full-access`, as the other
workers never ask.

- A turn ends at the response to `session/prompt`, so `send()` does not
  wait for it. A steer is a `session/prompt` with `_meta.fx.steer`. The
  wire is idle only when no request waits for a response; each message
  is accepted once, and one that fx refuses or drops goes again, in the
  order of the sends.
- `ai` tools reach fx through the bridge as an HTTP MCP server of the
  session. Their calls stay out of the stream.
- A new session with a past is written into the fx session store, with
  tool calls, results and images, and opened with `session/load`. The
  replay is the check: if fx refuses the session, the worker removes
  what it wrote and raises `WorkerError` with the fx version.
- A resume reads `session/list` first, because fx does not refuse a
  session of a different directory.
- `JsonRpcPeer` gets a private `_begin_call`: the id of a request, a step
  before the write, and a completion in read order. A failed write now
  removes its waiter.

Limits, stated on `Fx`: a bridge tool must answer in 5 minutes, streamed
tool results are short previews, instructions are fixed when the session
starts, and the built-in tools of fx cannot be limited. The store format
is internal to fx, so a new fx release can break the seed.

The live tests passed with fx 0.0.17
(`AI_WORKERS_LIVE=1 uv run pytest tests/workers/test_live.py -k fx`).
One new live test covers a session that starts with a past, for each
CLI.
Comments and docstrings only.

- _utils.py: the seeding rule says what fx does with reasoning (omits
  it) and with an image known only by URL (a note, as pi).
- _process.py: fx is one of the wires on `spawn` and `Process`.
- tests/workers/test_handoff.py: fx is one of the renderers that omit
  reasoning.
Comment thread examples/.test_scripts/run-examples.py Resolved
Comment thread examples/.test_scripts/run-examples.py Resolved
Comment thread examples/.test_scripts/run-examples.py Resolved
Comment thread examples/external/custom_loop.py Resolved
Comment thread examples/external/custom_loop.py Resolved
Comment thread examples/external/until_done.py Resolved
Comment thread examples/external/resume.py Resolved
Comment thread src/ai/external/claude_code/wire.py Resolved
The samples moved to `examples/external/`, but the list still had the
`workers/` paths, so the flag found no files.

- `WORKER_SAMPLES` is `EXTERNAL_SAMPLES`, and `--workers` is `--external`.
- `examples/README.md` names the folder and the flag.
`async with harness.session()` is now the only way to start the CLI. A
model call (`ai.stream`, `ai.Agent.run`) runs on the open session of its
external harness, and raises `ExternalHarnessError` when there is none.

- The call without a block is gone: no hidden session on the external
  harness (`_session`), no CLI that starts for one turn and stays for a
  tool loop, no copy overrides.
- An external harness has one open session in a context: a second
  `harness.session()` in the block raises. Two tasks can each open one.
- Core does not change.
- The examples and the tests open a session. The live test for calls
  without a session is gone.
An external harness is now settings only. The session holds the state:
`session.id` and `session.messages`.

- `harness.session()` is always a new session, and
  `harness.session(session_id)` continues one that the CLI has. To
  continue later, save `session.id` with the external harness.
- A model call finds its session by the external harness object. A copy
  of the external harness is another external harness.
- No external harness remembers a session, so a new conversation needs
  no `model_copy(update={"session_id": None})`. `custom_loop.py` loses
  that line, `resume.py` saves the session id, and `combine.py` uses one
  external harness with two sessions.
- Saved data with `session_id` does not load. No alias.
`start()` and `close()` were a second way to open a session. Now the
block is the only way: `start()` only starts the CLI early, or again
after it stopped, and raises outside `async with harness.session()`.

- A session that is open cannot be entered again.
- `close()` stays; `async with session` opens a closed session again.
`custom_loop.py` and `until_done.py` ran `subprocess.run` in a thread.
They now use `asyncio.create_subprocess_exec`, with the same 30 second
limit: on a timeout they kill the child and raise.
`ExternalHarness.model_validate_json(saved)` gives the saved subclass.
`resume.py` asserts it, and a test covers the JSON path for each of the
four classes. Before, only the live test loaded from JSON.
`import_optional_sdk` returns `types.ModuleType`, and `_sdk()` said `Any`.
@anbuzin anbuzin changed the title Implement ai.workers: Worker, WorkerProvider and Session Implement ai.external: ExternalHarness, ExternalHarnessProvider and Session Nov 17, 2026
You open the external harness, and you give the external harness to
`ai.stream`: the block is `async with harness`, not
`async with harness.session()`. One external harness is one
conversation. It comes with one session, `harness.session`, and it has
one live CLI at a time.

- `ExternalHarness` is a stdlib dataclass, not a pydantic model.
  `resume` and `history` are init-only: `Codex(resume=session_id)`
  continues a session that the CLI has, and `Codex(history=messages)`
  starts a new one with a past.
- The session has the state and the direct control: `id`, `messages`,
  `start`, `send`, `events` and `interrupt`. You do not make, open or
  close it. `harness.session()`, `async with session` and
  `Session.close()` are gone.
- A copy or a pickle is a closed external harness with the same
  settings and the same conversation. `dataclasses.replace(harness)`
  gives the same settings with a new session.
- The check for two external harnesses on one CLI session is for the
  process, not for one context.
- Saved settings are gone: no `model_dump()` and no class registry.
  `resume.py` saves the session id only. The provider stays a frozen
  pydantic model.
The examples make the external harness in the `async with` line:
`async with ai.external.Codex(cwd=tmp) as harness`. They use
`harness.session` directly, without a `session` variable.

- `agent_tools.py` uses Fx and `direct_control.py` uses Codex. The
  `KINDS` table and the command-line argument are gone.
- `custom_loop.py` and `combine.py` make a new external harness for
  each conversation. `until_done.py` takes an open external harness.
- For a new conversation, make a new external harness, not
  `dataclasses.replace(harness)`. The docstring says so, and the tests
  do not use `dataclasses.replace`.
- The durable agent examples keep the assertion on the model, without
  the TODO.
- `run-examples.py` lists `fx` with the CLIs that the external samples
  need.
@anbuzin
anbuzin marked this pull request as ready for review November 17, 2026 23:20
Comment thread src/ai/external/_bridge.py Resolved
Comment thread src/ai/external/claude_code/wire.py Resolved
Comment thread src/ai/external/_session.py Resolved
@anbuzin
anbuzin merged commit 501814b into main Nov 18, 2026
13 checks passed
@anbuzin
anbuzin deleted the workers-claude-code-pi branch November 18, 2026 00:13

This branch was successfully deployed

1 active deployment
Preview — bd44c9ae Deployed Nov 17, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature New functionality

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants