Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -34,8 +34,9 @@ jobs:
sudo apt-get install -y bubblewrap strace
python -m venv "$RUNNER_TEMP/test-env"
"$RUNNER_TEMP/test-env/bin/python" -m pip install --upgrade pip
# the methods' dependency groups, so their tests run rather than skip (not ace's: it pulls torch)
"$RUNNER_TEMP/test-env/bin/python" -m pip install -e '.[test]' \
# the methods' dependency groups and the mini-swe-agent extra, so their tests run rather than skip (not
# ace's group: it pulls torch)
"$RUNNER_TEMP/test-env/bin/python" -m pip install -e '.[test,mini-swe-agent]' \
--group gepa --group skillopt --group harness2 --group rrsi
echo "$RUNNER_TEMP/test-env/bin" >> "$GITHUB_PATH"
- name: The test groups this pull request touches
Expand Down
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,12 @@ Changes are grouped by software version. Entries under Unreleased describe work
call's tools are an MCP server offered directly; a failed request fails the session although Pi exits 0. Pi takes
no roles and has no web search; `image_input: true` declares a bound model that takes images. `hermit deploy
--host pi` declares the adapter's tools `exposure: direct`, with a 120 s timeout.
- mini-swe-agent as a CLI Base Agent (`mini-swe-agent`): the acting agent (`agent.name` or `agent.base:
mini-swe-agent`) runs one task through mini-swe-agent 2.4.6's Python API in a process of its own, whose JSON
events are the stream; a model reference is called through litellm with the key passed per request, and the
agent's commands get only `env`. The call's tools are one bash command, `hermit-tool`. `hermit deploy --host
mini-swe-agent` writes `mini-swe-agent.yaml`. Installed with the `mini-swe-agent` extra.
[Tutorial](docs/tutorials/mini-swe-agent.md).
- Models in sandboxes as Harbor's agents get theirs: each sandbox gets its models' key variables and provider hosts
(an isolated network becomes an allowlist) and forwards a model meter on the loopback; what a run keeps (the
learning material, the step records, the adapter) is scrubbed of key values (`hermit/redact.py`). The allowlist
Expand Down
13 changes: 13 additions & 0 deletions docs/_examples/mini_swe_agent/environment/inventory.csv
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
item,warehouse,units
bolt,north,120
bolt,south,75
nut,north,300
nut,east,45
washer,south,80
washer,east,20
bracket,north,12
bracket,south,30
hinge,east,64
hinge,north,16
spring,south,9
spring,east,41
31 changes: 31 additions & 0 deletions docs/_examples/mini_swe_agent/judge.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
"""Private answers are kept in the judge, outside learner-visible task rows."""
import json

from hermit.task import JudgeResult, RunOutput

ANSWERS = {
"train-001": 195,
"train-002": 170,
"train-003": 4,
"val-001": 345,
"val-002": 194,
"val-003": 6,
"test-001": 80,
"test-002": 448,
"test-003": 5,
}


def score(task: dict, output: RunOutput) -> JudgeResult:
expected = ANSWERS[task["id"]]
try:
value = json.loads(output.output)
except (ValueError, TypeError):
value = None
correct = isinstance(value, dict) and value == {"answer": expected}
feedback = "Correct." if correct else (
'Return only a JSON object with exactly one key, "answer", whose value is an integer, for example '
'{"answer": 12}. No Markdown, units or explanation. '
f"The reference answer for this attempt is {expected}."
)
return JudgeResult(score=float(correct), feedback=feedback)
37 changes: 37 additions & 0 deletions docs/_examples/mini_swe_agent/run.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
# mini-swe-agent as the acting agent: nine questions about a CSV file, answered with bash commands. See
# docs/tutorials/mini-swe-agent.md. One model, a `<backend>/<model id>` reference (hermit backends), acts and learns.
model: deepseek/deepseek-flash
data:
tasks: tasks.jsonl
train: train.txt
val: val.txt
test: test.txt
step_size: 3
shuffle: false
task_info: task-info
judge:
path: judge.py
template: prompt-only@1.0
evolver:
name: gepa
params:
max_tokens: 4096
sandbox:
enforce: best_effort # bubblewrap where it is installed; elsewhere unconfined, recorded and warned
environment:
read: [environment] # the task's file, copied into each workspace; the data, the judge and the runs stay out
trainer:
run_name: mini-swe-agent
output_dir: runs
epochs: 1
budget:
tokens: 600000
hours: 0.5
call_timeout_min: 10
agent:
name: mini-swe-agent
concurrency: 3
params:
files: [environment/inventory.csv]
web_search: false # set either way for every CLI agent; mini-swe-agent has none
timeout_s: 300
8 changes: 8 additions & 0 deletions docs/_examples/mini_swe_agent/task-info/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
# Inventory questions

Each task asks one question about `inventory.csv`, a small table of items, warehouses and unit counts that is copied
into the agent's working directory. The agent answers by running bash commands, mini-swe-agent's only action.

The judge accepts exactly `{"answer": <integer>}`; after a wrong or malformed answer its feedback gives the format
rule and the reference answer. Learn the output format from training feedback. Test tasks are held out from the
learner.
9 changes: 9 additions & 0 deletions docs/_examples/mini_swe_agent/tasks.jsonl
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
{"id":"train-001","prompt":"How many units of bolt are there across all warehouses in inventory.csv?"}
{"id":"train-002","prompt":"How many units are stored in the east warehouse in inventory.csv?"}
{"id":"train-003","prompt":"How many rows of inventory.csv have fewer than 25 units?"}
{"id":"val-001","prompt":"How many units of nut are there across all warehouses in inventory.csv?"}
{"id":"val-002","prompt":"How many units are stored in the south warehouse in inventory.csv?"}
{"id":"val-003","prompt":"How many distinct items does inventory.csv list?"}
{"id":"test-001","prompt":"How many units of hinge are there across all warehouses in inventory.csv?"}
{"id":"test-002","prompt":"How many units are stored in the north warehouse in inventory.csv?"}
{"id":"test-003","prompt":"How many rows of inventory.csv have more than 60 units?"}
3 changes: 3 additions & 0 deletions docs/_examples/mini_swe_agent/test.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
test-001
test-002
test-003
3 changes: 3 additions & 0 deletions docs/_examples/mini_swe_agent/train.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
train-001
train-002
train-003
3 changes: 3 additions & 0 deletions docs/_examples/mini_swe_agent/val.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
val-001
val-002
val-003
2 changes: 1 addition & 1 deletion docs/faq.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ Follow [Deploy an adapter](tutorials/deploy-an-adapter.md) for the commands and
<details class="ha-faq" id="can-i-reuse-an-adapter-across-models-and-harnesses">
<summary>Can I Reuse an Adapter Across Models and Harnesses?</summary>

Yes, when the receiving agent supports the exports the adapter needs. Hermit currently generates workspaces for Codex, Claude Code, and Pi, and converts native plugin packages for Claude Code, Codex, Pi, and Hermes. Each route has different capabilities; a target may refuse unsupported exports.
Yes, when the receiving agent supports the exports the adapter needs. Hermit currently generates workspaces for Codex, Claude Code, Pi, and mini-swe-agent, and converts native plugin packages for Claude Code, Codex, Pi, and Hermes. Each route has different capabilities; a target may refuse unsupported exports.

Compatibility does not guarantee the same performance on a different model, harness, or task. Evaluate the adapter in its new setting. See the [host capability matrix](framework/commands.md#what-each-hosts-plugin-carries) and [compatibility limits](reference/compatibility.md).

Expand Down
28 changes: 18 additions & 10 deletions docs/framework/adapter.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,18 +76,18 @@ id>/`. Paths in the instructions below, and in the adapter's skills and roles, a
command they give from there." Then each export `exports.yaml` declares is
connected to the host's own place when the host has one, so that the host loads it; an export the host has no place
for stays a file of the copy, and the context names it with what to do with it. The rule is the same for every
host, and for the built-in `codex` and `claude-code` acting agents, which deploy as `hermit deploy` does: an adapter
host, and for the built-in `codex`, `claude-code`, `pi` and `mini-swe-agent` acting agents, which deploy as `hermit deploy` does: an adapter
is deployed as far as the host can take it, and nothing it holds is out of the agent's reach.

| Adapter source | Deployment operation | Codex output | Claude Code output | Pi output |
|---|---|---|---|---|
| Every file but `.hermit/` and `.git/` | Copy it as it is. | `hermit/<id>/` | `hermit/<id>/` | `hermit/<id>/` |
| `ADAPTER.md` (always exported) | Insert the line above and its text into a managed instruction block. | `AGENTS.md` | `CLAUDE.md` | `AGENTS.md` |
| `skills/verify-output/`, through `skills` in `exports.yaml` | Copy the skill directory. | `.codex/skills/verify-output/` | `.claude/skills/verify-output/` | `.pi/skills/verify-output/` |
| `agents/reviewer.md` (or `agents/reviewer/AGENT.md`), through `agents` in `exports.yaml` | Convert the subagent into native host files: frontmatter gives name and description, the body becomes its instructions. | `.codex/agents/reviewer.toml`, registered in `.codex/config.toml` | `.claude/agents/reviewer.md` | named in `AGENTS.md` (Pi has no subagents): name, description, the path of its instructions |
| `tools.py`, through `tools` in `exports.yaml` | Configure a framework MCP server that loads the source adapter. | `.codex/config.toml` | `.mcp.json` | `.pi/mcp.json` |
| a hooks file, through `hooks` in `exports.yaml` | Write the hooks the host runs, each command run from `hermit/<id>/`; name the others in the context (a command to run, a `prompt` hook's check to make at that moment); a hook of another type is refused. | `.codex/hooks.json` (command hooks; Codex runs no prompt or agent hooks) | `hooks` in `.claude/settings.json` (command and prompt hooks) | named in `AGENTS.md` (Pi reads no hook file) |
| every other file of the copy | Name it, with its frontmatter description or first line, under "Adapter resources" in the context (up to 60 files; the rest are counted). | `AGENTS.md` | `CLAUDE.md` | `AGENTS.md` |
| Adapter source | Deployment operation | Codex output | Claude Code output | Pi output | mini-swe-agent output |
|---|---|---|---|---|---|
| Every file but `.hermit/` and `.git/` | Copy it as it is. | `hermit/<id>/` | `hermit/<id>/` | `hermit/<id>/` | `hermit/<id>/` |
| `ADAPTER.md` (always exported) | Insert the line above and its text into a managed instruction block. | `AGENTS.md` | `CLAUDE.md` | `AGENTS.md` | the system template of `mini-swe-agent.yaml` |
| `skills/verify-output/`, through `skills` in `exports.yaml` | Copy the skill directory. | `.codex/skills/verify-output/` | `.claude/skills/verify-output/` | `.pi/skills/verify-output/` | named in the system template (mini-swe-agent loads no skills): name, description, the path of its `SKILL.md` |
| `agents/reviewer.md` (or `agents/reviewer/AGENT.md`), through `agents` in `exports.yaml` | Convert the subagent into native host files: frontmatter gives name and description, the body becomes its instructions. | `.codex/agents/reviewer.toml`, registered in `.codex/config.toml` | `.claude/agents/reviewer.md` | named in `AGENTS.md` (Pi has no subagents): name, description, the path of its instructions | named in the system template (no subagents) |
| `tools.py`, through `tools` in `exports.yaml` | Configure a framework MCP server that loads the source adapter. | `.codex/config.toml` | `.mcp.json` | `.pi/mcp.json` | `environment` in `mini-swe-agent.yaml`: Hermit's environment class answers the `hermit-tool` command |
| a hooks file, through `hooks` in `exports.yaml` | Write the hooks the host runs, each command run from `hermit/<id>/`; name the others in the context (a command to run, a `prompt` hook's check to make at that moment); a hook of another type is refused. | `.codex/hooks.json` (command hooks; Codex runs no prompt or agent hooks) | `hooks` in `.claude/settings.json` (command and prompt hooks) | named in `AGENTS.md` (Pi reads no hook file) | named in the system template (mini-swe-agent runs no hooks) |
| every other file of the copy | Name it, with its frontmatter description or first line, under "Adapter resources" in the context (up to 60 files; the rest are counted). | `AGENTS.md` | `CLAUDE.md` | `AGENTS.md` | the system template |

This is how the adapter reaches the acting system. The host reads the generated instruction, skill, role and hook
files; tool configuration makes the declared tools available through its tool interface. What the adapter does not
Expand Down Expand Up @@ -140,6 +140,14 @@ pi-workspace/ # only for an adapter without roles
.hermit-deploy.json
```

```text
mini-swe-agent-workspace/
mini-swe-agent.yaml # the context, the resources and the tools in its system template
hermit/file-example/
ADAPTER.md exports.yaml tools.py skills/ agents/ README.md
.hermit-deploy.json
```

Only declared exports are connected; every file is copied. A prompt-only adapter has no tool-server entry, skills,
roles or hooks to install, and its copy holds `ADAPTER.md`, `exports.yaml` and `README.md`. The generated deployment
report identifies the actual paths and role names, and lists every copied file with its digest.
Expand Down
17 changes: 11 additions & 6 deletions docs/framework/agent-layers.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ How a model takes steps. Layer 2 holds only the framework's agents:
| `codex` | The Codex CLI, working in a workspace |
| `claude` | The Claude Code CLI, working in a workspace |
| `pi` | The Pi CLI, working in a workspace (no roles, no web search) |
| `mini-swe-agent` | mini-swe-agent's bash loop, working in a workspace: one task through its Python API, in a process of its own (no roles, no web search) |

API, the only one Layer 3 sees:

Expand All @@ -69,7 +70,7 @@ agent, prompt = agent.instructed(system, prompt) # a copy that takes `system`
environment, step by step in a resumed learning session. Codex and Claude Code resume their thread; direct and
react carry the message history.
- `roles`: named sub-agents (name, description, instructions) the agent may delegate to: Codex's roles, Claude
Code's agents. direct and react refuse them.
Code's agents. direct, react, pi and mini-swe-agent refuse them.
- `trajectory`: the run as standard steps: messages, reasoning, commands with their exit codes and output, tool
calls, file changes, errors. Layer 3 reads only these; a CLI's raw log (`agent-codex.jsonl`) is kept beside them
as an archive.
Expand All @@ -81,10 +82,12 @@ none), Codex's own sandbox mode, the CLI's home, the time limit (the full list i
leaves the agent as it was. Running a task again when the provider could not serve it is Layer 3's: the workspace
is placed anew first.

Codex, Claude Code and Pi share one runner, `CliAgent`: the isolation, the home, the sign-in, the tool server and the
process are the same for every CLI. Each CLI is a subclass whose hooks give what is its own: its command, its home,
and how its event stream is read (the reply, the trajectory, the usage, whether the provider served it, and what a
session that was not served met: `outcome`). A new CLI is
Codex, Claude Code, Pi and mini-swe-agent share one runner, `CliAgent`: the isolation, the home, the sign-in, the tool
server and the process are the same for every CLI. Each CLI is a subclass whose hooks give what is its own: its
command, its home, and how its event stream is read (the reply, the trajectory, the usage, whether the provider served
it, and what a session that was not served met: `outcome`). mini-swe-agent's process is the framework's own
(`hermit/agent/base/mini_swe_agent_run.py`): it runs one task through mini-swe-agent's Python API and writes the
session as JSON events, the stream the runner reads. A new CLI is
a subclass listed in `CLIS`, a reader of its stream in `hermit/agent/base/trajectory.py`, and its model binding in
Layer 1.

Expand Down Expand Up @@ -142,6 +145,7 @@ Each Base Agent offers the tools to the model in its own way:
| `direct` | Their schemas in the API request | Layer 3, which then calls `direct` again |
| `react` | Their schemas in the API request | `react`, in its loop |
| `codex`, `claude`, `pi` | The framework's tool server, over MCP (streamable HTTP) | The tool server |
| `mini-swe-agent` | One bash command, `hermit-tool <name> '<JSON arguments>'`, listed in its system message | The tool server, through its JSON entrance |

The tool server (`hermit/agent/base/tool_server.py`) is started for the one call and stopped after it. It carries a
token in its URL, so that only the session it was started for can call it, and tells Codex to approve its tools
Expand All @@ -156,7 +160,8 @@ inside the agent's sandbox with the agent's permissions.
## Limits

`limits` states what the agent may touch. Each Base Agent enforces it with its own isolation, and refuses a limit it
cannot enforce (`Unsupported`) rather than ignore it. For Codex and Claude Code:
cannot enforce (`Unsupported`) rather than ignore it. For Codex, Claude Code, Pi and mini-swe-agent (Pi and
mini-swe-agent run in the run's sandbox or with no isolation, not in a container):

| Limit | In a container | In the run's sandbox | No isolation |
|---|---|---|---|
Expand Down
6 changes: 5 additions & 1 deletion docs/framework/commands.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,12 +71,16 @@ user chooses: `--adapter none` runs the acting agent with no harness adapter at
hermit deploy --adapter DIR --workspace WORKSPACE --host codex
hermit deploy --adapter DIR --workspace WORKSPACE --host claude-code
hermit deploy --adapter DIR --workspace WORKSPACE --host pi
hermit deploy --adapter DIR --workspace WORKSPACE --host mini-swe-agent
```

This generates host instruction files, role/skill files, and tool configuration. It does not launch the acting
agent. Pi has no subagents and reads no hook file, so `--host pi` names an adapter's roles and hooks in `AGENTS.md`
instead; its `.pi/` files (tools, skills) load once the user trusts the project in Pi, while `AGENTS.md` needs no
trust. `--context-limit BYTES` (Codex) is the `project_doc_max_bytes` your Codex configuration sets, 32768 (Codex's
trust. mini-swe-agent reads one config file, so `--host mini-swe-agent` writes `mini-swe-agent.yaml`: Hermit's
prompts over mini-swe-agent's `mini` config, with the context, the names of the adapter's skills, roles, hooks and
other files, and its tools as one bash command (`hermit-tool`) in the system template; run `mini -c
mini-swe-agent.yaml` from the workspace ([tutorial](../tutorials/mini-swe-agent.md)). `--context-limit BYTES` (Codex) is the `project_doc_max_bytes` your Codex configuration sets, 32768 (Codex's
default) when not given: a longer `AGENTS.md` is refused, since Codex would cut it; `0` skips the check. It is the per-project way to use an adapter; the per-agent way, `convert`, and how the two compare are in
[What is a harness adapter?](what-is-a-harness-adapter.md#two-ways-to-use-an-adapter). The workspace must be an empty directory or one previously generated for the same host, separate from
the adapter directory. See the [file mapping](adapter.md#acting-interface) for exact paths and update behavior.
Expand Down
Loading