Skip to content

Add DeepSeek Harness (dsh) as a CLI Base Agent (acting agent and dsh Evolver) - #90

Draft
iamziyuzhao wants to merge 12 commits into
mainfrom
feat/dsh
Draft

iamziyuzhao wants to merge 12 commits into
mainfrom
feat/dsh

Conversation

@iamziyuzhao

@iamziyuzhao iamziyuzhao commented Nov 9, 2026 •

Copy link
Copy Markdown
Collaborator

Change

Adds DeepSeek Harness (dsh; npm @deepseek-ai/dsh 0.2.2-rc.1,
deepseek-ai/deepseek-harness) as a fourth CLI Base Agent beside
codex, claude and pi. It is one CliAgent subclass that reports its runs through the runner's outcome()
(#77). It can act (agent.base: dsh) and learn (the new dsh Evolver). It follows #89 (Pi).

Based on main (d5f0a64, after #89 merged). The first version of this PR was stacked on #89's earlier head. The
branch was rebuilt on main, so its diff is now only dsh: 50 files, +2394 −100. Draft: the documentation is not
written yet (see Docs).

  • Layer 1: a <backend>/<model> reference becomes a route of dsh's pi-ai adapter (Binding.dsh_provider): the
    backend's URL in the first dialect it serves (as for Pi: chat completions, then Messages, then Responses), the key
    by its variable's name (apiKeyEnv, never the value) and the context window. dsh reports no cost, so the backend's
    price is applied to the usage it reports.
  • Layer 2: Dsh(CliAgent) runs dsh --profile headless --json in a private home (DSH_HOME, HOME,
    DSH_AGENTS_HOME), with telemetry off and the caller's own DSH_*, DEEPSEEK_BASE_URL and
    DEEPSEEK_SEARCH_BASE_URL dropped (without the sandbox they moved dsh's own provider and its web search), whatever
    a sandbox lets through: hermit's own DEEPSEEK_BASE_URL is its deepseek backend's chat completions root, which
    dsh reads as its own provider's Messages root. The session's env gives dsh one of its own.
    • Settings: the session's settings are a patch layer of dsh's own rows (--patch, written in the home):
      • the bound model as the default model (effort as its reasoningEffort; image_input: true declares a model
        that takes images);
      • the call's tools as an MCP server, offered to the model as mcp__<server>__<tool>;
      • web search and fetch off unless web_search: true. dsh's search is DeepSeek's own service (its Messages API
        on DEEPSEEK_API_KEY), so unless the model is reached there on that key the session gets web fetch only, as
        Codex's hosted search is off for a backend other than OpenAI's;
      • the session's transcripts in plain JSON lines (dsh compresses them with zstd), so that hermit.redact scrubs a
        key a session wrote from its home, as from every other record;
      • off in every session: the title request, telemetry (also DSH_TELEMETRY_DISABLED=1), the session log and
        plugin list that dsh adds to requests to DeepSeek's own API, and sub-agents and workflows (their tokens are in
        no usage figure dsh reports).
    • The workspace is the session's project, wherever it lies. Two rows of the layer make it so: skills come only
      from the workspace's .dsh/skills (skill-filesystem), and instructions only from the working directory, the
      workspace (agent-instructions with no root marker). Without them, a workspace inside a git repository also got
      that repository's skills and AGENTS.md, which outranked the adapter's. The dsh Evolver's sessions take no
      instruction file at all (instruction_files: false): dsh adds a folder's AGENTS.md when its file tools touch
      the folder, and an adapter's AGENTS.md is the acting agent's policy, which Codex's learner is never given.
    • Task data: a task's data inside the workspace is bound there by bubblewrap, a real directory, as for Claude
      Code (binds_mounts): dsh's glob and grep run ripgrep without --follow, so a link was never searched.
    • Keys: a backend that needs no key gets the placeholder key local through a variable of the session
      (HERMIT_DSH_NO_KEY), because dsh refuses every request of a route that names no key.
    • Sandbox: commands run in dsh's own sandbox (agent.dsh.permission_mode, default workspace-write).
      writable paths are granted by their real paths, as dsh requires. Approvals are never asked
      (--approval-policy never). A permission mode or an effort level dsh does not have is refused before any
      session: the acting agent's as the run config loads, the dsh Evolver's by hermit check and before train's
      first step, from its params as the Trainer has them (the method's own and the run's model in). dsh's own provider
      takes fewer levels (off, low, high, max) than a bound model's route. YAML's bare off, a boolean, is
      off.
    • Resume by --session-id when the home keeps the session (dsh resumes only in the directory the session ran
      in, which an AgentEvolver's calls share).
    • Instructions: dsh has no system prompt for one run, and its persona text fails a step on any {{...}} other
      than {{provider}} and {{model}}, so system instructions go at the top of the prompt as a marked block.
    • Refused: roles, a login, and container isolation.
    • outcome():
      • A run without a completed final event has failed, whatever dsh's exit code (dsh exits 0 when SIGTERM stops
        it).
      • So has a run whose tools did not start: dsh only warns on stderr and runs on without them.
      • A run that wrote no session event (a usage error, or a working directory's .env that sets a name dsh
        keeps, such as PATH) started no session (started: false), so the acting agent reports it instead of
        recording a wrong answer for every task.
    • Usage is read from status step_end events (records.parse_dsh), each the sum of the step's attempts. A
      step that produced something but whose tokens dsh did not report (the provider reported no usage, or an attempt
      reported none and dsh left the step's usage out) is an unreported call, and the session's cost is then unknown
      rather than the price of the tokens reported. A step dsh retried after a refused request loses its cache counts
      (dsh keeps a count only when every attempt reported one), which its totalTokens still holds: those tokens are
      counted as input and priced as hermit prices a cache write, never less than the step's price with the counts
      (for a price whose cached input is no dearer than its input), so a USD budget still counts the session. On the
      Responses API the total is the server's own total_tokens, taken as given.
    • Context window: a run that ends CONTEXT_WINDOW_EXCEEDED with a backend that declares no context_window
      says to declare it. dsh sizes a model it does not know as 1,000,000 tokens with 128,000 output tokens a request,
      which a vLLM server refuses; with the window declared, dsh sizes each request by it.
    • Trajectory from the stream (trajectory.dsh_steps). A command's exit code is read from dsh's own end
      markers, anchored at the end as dsh reads them. It is unknown for a command that reached its timeout (in a
      session dsh moves it to a background job, or stops it when it cannot; the timeout printed is the one dsh applied:
      the model's, 1000.5 too, else 60 s, at most 600 s), was stopped, was killed by a signal or runs in the
      background, and for a result dsh's --json cut at 8 KiB (the marker is in the cut part; the step says it was
      cut). Every unfinished end (error, aborted, max-tokens, blocked, ...) is an error step.
  • Layer 3:
    • hermit/hosts/dsh.py (hermit deploy --host dsh) writes AGENTS.md, .dsh/skills, and a patch layer,
      .dsh/hermit.patch.yml, because dsh has no project file for MCP servers or skill folders. The layer holds the
      adapter's tool server (an insert row, 120 s per call, which fails the session's start when dsh cannot start
      it, as Codex's required does) and names .dsh/skills. A context past what dsh gives the model of a project's
      instructions (65,536 bytes, its 304-byte frame around AGENTS.md included) is refused. The acting agent passes
      the layer with --patch.
    • AgentEvolver, the loader and the config accept dsh. methods/dsh/ is the new Evolver. It turns dsh's own
      sandbox off inside the evolver sandbox (permission_mode: danger-full-access), as Codex's is, and copies no
      login (home: []).
    • The AHE template runs its hooks by hand on dsh, as on Pi (hook_hosts.py gains a dsh wire).
  • Login: a model reference. Without one, dsh uses its own provider (deepseek-official), and the user's
    ~/.dsh/.env (or $DSH_HOME/.env) is copied into the session's home, as Codex's auth.json is. This path was not
    run (see Validation).
  • Core changes (shared code):
    • UNAVAILABLE gains dsh's codes for a provider that could not serve (RATE_LIMIT, SERVER, TIMEOUT,
      TRANSPORT, EMPTY_RESPONSE, STREAM_CLOSED), so the acting agent runs the task again. They match only at the
      start of a dsh: <CODE>: line.
    • The runner (CliAgent.run): outcome() may now say started. When it does not, started is as before (the CLI
      wrote a JSON line).
    • records.parse_log reads a dsh stream with dsh's reader. It read every stream other than Codex's as Claude
      Code's (a Pi stream too: fixed in Read a Pi acting agent's event stream as Pi's in parse_log and the AHE Evolver's texts #91).
    • Usage gains unreported_calls, the calls whose tokens a CLI did not report, and records.price_usage leaves
      such a usage's cost unknown (the budget counts it as such). Only dsh's reader sets it: Codex, Claude Code and Pi
      are priced as on main.
    • The AHE Evolver's texts name a dsh acting agent's stream (agent-dsh.jsonl) instead of agent-codex.jsonl
      (Pi's is missing on main too: fixed in Read a Pi acting agent's event stream as Pi's in parse_log and the AHE Evolver's texts #91).
    • The mwm and RRSI Evolvers refuse dsh at setup: mwm would run it on dsh's own provider instead of its bound
      model (its session settings carry no provider), and RRSI's read-only roles would run with all of dsh's tools
      (their settings are Codex's or Claude Code's). pi has the same problem on main: mwm and RRSI: refuse a learner backend other than Codex or Claude Code at setup #92 refuses
      every backend but codex and claude, which covers dsh too.
    • login.evolver_problems, which train runs before its first step, also checks a dsh Evolver's settings
      (cli.dsh_problems), and hermit check now reports it for an Evolver without a login too (it did only with one).
    • hermit/isolate.py: only its config check (validate). dsh refuses a login, as Pi does, and the acting agent's
      dsh settings are checked there, as Claude Code's effort is on a login. No sandbox mechanics change (.dsh is a
      one-level folder). Pending: the core team's agreement, which the sandbox checklist asks for any isolate.py
      change.
  • Docs: not written yet (their wording is being chosen). The plan follows Add Pi as a CLI Base Agent (acting agent and pi Evolver) #89 (agent-layers.md,
    model-backends.md, config-fields.md, task.md, evolver.md, methods/agent.md, custom-evolver.md,
    compatibility.md, installation.md, CHANGELOG), plus commands.md, adapter.md and methods/ahe_adapter.md for
    hermit deploy --host dsh and the AHE hooks.
  • Not included: hermit convert --target dsh (an adapter as a dsh plugin package).

Validation

Run in an Ubuntu 24.04 container (arm64), as root with --privileged because the VM's AppArmor setting refuses
unprivileged user namespaces. Python 3.12, bubblewrap, Node 24.22.2, and dsh 0.2.2-rc.1 from npm (its pi-ai
dependency resolved to 1.5.0). The toy runs ran on 207ea88. After it, 1bbd57c adds one test (and a docstring's
line breaks); f4792e0 a recorded stream (promoted.jsonl) and, in an existing test, its case and two
[stopped: …] results written by hand as dsh's bash tool renders them; d25958a widens the two timeout markers'
patterns (round 8) and records promoted.jsonl and bash.jsonl again with the calls its tests read; a43698d
corrects wording only (round 9: comments, a docstring, the fixture README). The full suite ran again on the head,
a43698d.

Check Result
python -m pytest --integration, main (d5f0a64) 2 failed, 1164 passed, 85 skipped
The same with the rrsi group installed, main 2 failed, 1229 passed, 80 skipped
The same, this branch (a43698d) 2 failed, 1276 passed, 80 skipped
tests/core/test_dsh_agent.py (41 tests) and test_deploy.py::test_dsh* All pass, including the 11 that run the installed dsh
hermit train + hermit eval on the docs' toy task, sandbox enforced, dsh acting and learning, scripted model hermit check ok; train 0.0 → 1.0; eval: final adapter 1.0, none 0.0; no call of unknown cost
The same with DeepSeek V4.1 Flash on Fireworks, reached through the sandbox's allowlist proxy train 0.0 → 1.0 in 2 steps; eval: final adapter 1.0, none 0.0; $0.058 in all as hermit prices it (training, both evals and the check's probe task), no call of unknown cost; the learner's sandbox allowed only api.fireworks.ai, and its probe found the network and the run's data closed; the key in no file the run wrote
  • Failures: the same 2 on both trees, which fail when run as root:
    tests/core/test_isolate.py::test_a_rollout_sees_what_is_declared_and_nothing_else and
    tests/methods/ahe/test_ahe_hook_runner.py::test_logs_that_cannot_be_written_lose_the_line_not_the_call.
  • Fixtures (tests/fixtures/dsh/) are dsh's output unchanged, recorded as their README says.
  • dsh's behaviour was checked against its source at the tag dsh-v0.2.2-rc.1 and with the installed dsh.
  • Not verified:
    • macOS. dsh's sandbox there is Seatbelt, which cannot nest inside another Seatbelt sandbox.
    • dsh's own bubblewrap sandbox nested in hermit's as an unprivileged user. dsh then tries Landlock; if neither
      starts, every command fails with SANDBOX_UNAVAILABLE, visibly.
    • dsh's own provider with ~/.dsh/.env.
    • A backend that serves only the Messages or the Responses API.
Known dsh behaviours (not changed here)
  • Usage dsh does not report:
    • a compaction's summary request;
    • web searches: with web_search: true on DeepSeek's API, each search is a request of its own on the same key;
    • the tokens of a step one of whose attempts reported none (dsh then leaves the step's usage out; it is counted
      as a call of unknown tokens);
    • what a provider billed for an attempt cut off mid-stream that reported zeros;
    • the last step of a session a stop (a timeout, a cancelled call) cut short before its end: dsh writes a step's
      usage at its end, after its tool calls ran, and a stopped session writes nothing more. Its cost is then that of
      its earlier steps; a session stopped in its first step reports no usage, so the acting agent's record has an
      unknown cost.
  • A task run again after an outage (unavailable) keeps only its last attempt's usage, for every CLI: the
    acting agent's retry loop on main reports the last result's usage. dsh's outage codes reach that loop, and a dsh
    session can fail with one after billed steps. A separate change (it is shared code).
  • A run started on an earlier commit of this branch wrote compressed transcripts, and dsh refuses a home that
    holds both kinds: such a run cannot be resumed on this head.
  • An Evolver with no model reference (dsh's, as Pi's on main) fails each step: dsh's own provider is not
    reached from the evolver sandbox.
  • .env files: dsh also reads a .env in the working directory. One that sets a reserved name (PATH, HOME,
    a proxy, DSH_*) stops dsh before it writes any event; the acting agent reports that the session did not start.
  • The deployed patch layer is a file of the workspace, so the agent can edit it for a later session, as it can
    Codex's .codex/config.toml.
  • Context window: see Change. With a backend that declares none, a vLLM server refuses every request; Fireworks
    caps the output tokens instead, and compaction starts late on a small-context model.
  • Nested instructions: dsh adds a folder's AGENTS.md when its file tools touch the folder. The acting agent's
    adapter copy (hermit/<id>/) holds one, identical to the context in the shipped templates.
  • dsh's own sandbox: with neither bubblewrap nor Landlock, dsh refuses each command (SANDBOX_UNAVAILABLE) and
    the session still answers. With a task's data bound, SIGTERM ends the binding's bubblewrap, and dsh with it.
  • Resume works only in the directory the session ran in.
Known issues from review rounds 7 and 9 (low severity, not changed)
  • Refused at each session rather than as the config loads (a misconfiguration; each task then fails with dsh's
    own message before any request): Claude Code's claude_permission_mode in a dsh agent's params becomes dsh's
    permission mode; a provider route given with no model.
  • DSH_HOME with a leading ~ (as export DSH_HOME="~/x" leaves it): dsh expands it, hermit does not, so the
    user's .env is not copied for dsh's own provider, whose sessions then lack the key.
  • A kept home's sessions folder replaced by a link (by the session itself) is still resumed through it, from
    where the session could read anyway.
  • Correct today, untested: agent.base: dsh selecting dsh by its name; the dispatch of
    hermit deploy --host dsh; agent: {name: cli, base: dsh} binding its model as dsh's; the dialect hermit check
    and train require of a dsh model's backend (needs()); a one-key override of a dsh section, and agent.name: dsh
    with another agent.base refused; the patch layer written past a link a session left in a kept home; a tool
    server name dsh refuses; a --patch=<path> in args; the clamp of a Responses total below the counts; a cut
    message, thought or tool result marked as cut; a tool other than bash, write, edit or an MCP tool kept as a
    tool-call step; the outage codes matched only at a line's start; the key half of the web-search rule;
    effort: off turning thinking off on dsh's own provider; (round 9) the exponent form of a timeout in both timeout
    markers (1e-7ms: JavaScript prints a timeout under 0.000001 ms so), which round 8's patterns match and which
    read as exit 0 before them; only the decimal form (1000.5ms) is tested.
  • Wording, in two test docstrings: one compares the acting agent's check at load with Claude Code's effort,
    which is checked at load only on a login; one says only dsh's own provider can turn thinking off (dsh's pi-ai
    routes can too; hermit declares none for a bound model).

Review

  • Tests and documentation cover the affected behavior, where applicable. Tests: see Validation.
    Documentation: pending.
  • Commands, defaults, and public interfaces are consistent. New: agent.base: dsh (agent.name: dsh),
    evolver.base: dsh, the dsh Evolver, a dsh: settings section (permission_mode, image_input, ...),
    hermit deploy --host dsh, and the provider setting for dsh. hermit check passes on a dsh config.
  • Shared logs and examples contain no credentials or private data. The patch layer holds the key's variable
    name, never its value. A grep found the Fireworks key in no record of the real run. The fixtures hold no key.
  • Documentation-only: N/A.
  • Core changes: see Change.
  • Failures reported accurately: tests cover a failed request (an outage sets unavailable, so the task is
    run again), a run stopped by SIGTERM (dsh exits 0), a run cut at the token limit, tools that did not start, a run
    that started no session, a command moved to a background job at its timeout or killed at it (also at a timeout of
    1000.5 ms), stopped, run in the background or cut (exit code unknown), and refused roles, login and container.
  • Real entry points: make("dsh").run, the acting agent's run (with the deployed adapter's own stdio tool
    server), hermit train and hermit eval. The stand-ins are only at the edges (the CLI, the model).
  • Test checklist: the dsh tests are in tests/core/test_dsh_agent.py (and test_deploy.py, the AHE, mwm
    and RRSI groups for their parts), named for the behaviour each holds; tests/fixtures/dsh/ is listed in
    tests/selection.py and tests/README.md, and its README says how each stream was recorded (dsh's output
    unchanged). Each review round's fixes were undone one at a time to see their test fail (round 8's: the decimal
    form of a timeout, not its exponent form; see Known issues). The stand-ins are a fake
    dsh that replays a recorded stream and a scripted model on the loopback; tests that run the installed dsh are
    marked integration and skip without it or without bubblewrap. Tests write to tmp_path, and the check under
    bubblewrap to a directory of its own under ~/.cache/hermit-tests, removed after it.
  • Sandbox checklist: every dsh session runs through hermit.sandbox: the acting agent in the task sandbox
    (isolate.py's worker), the dsh Evolver in the Trainer's evolver sandbox (a record from the Fireworks run:
    sandbox/learner-1.json, enforced: true, network isolated to api.fireworks.ai, its probe passed). dsh's own
    sandbox runs nested inside the run's for the acting agent (workspace-write), and is off inside the evolver
    sandbox, as Codex's is. The sandbox itself is unchanged; isolate.py changes only in its config check (see Core
    changes), which the checklist counts as a core change: pending the core team's agreement.
  • Review rounds before this update (fresh agents, separate from the author's session; in rounds 1-6 each
    was given the general, test and sandbox checklists):
    • Round 1, on 03fafff: two reviewers, 23 findings. Fixed in 7a076fb, each with a test that fails without
      its fix: bash results cut at 8 KiB read as exit 0; markers anchored with Python's $; background commands read
      as exit 0; tools that did not start went unnoticed; started true for a run that wrote only an error;
      --add-dir broke every command for a path through a link; agent.dsh.permission_mode was dropped; a git
      repository's skills and AGENTS.md reached an unsandboxed session; a workspace below a repository's root lost
      its deployed skills; no check of the 64 KiB instruction cap; usage of a step that answered without usage was not
      counted; an effort without a model was dropped; user_home() ignored DSH_HOME; AHE had no dsh wire; mwm and
      RRSI accepted backends they cannot run; parse_log read Pi's and dsh's streams as Claude Code's; tests for
      dsh's sandbox inside the run's, the context-tool portability loop and the deployed server's sandbox variables;
      docstrings. Explained, not changed: what dsh does not report (compaction), the agent-writable patch layer, resume
      in the same directory, and the .env stop (now reported as no session). Docs: pending.
    • Round 2, on 7a076fb: two reviewers, 20 findings and questions (three found by both). Fixed in 9d87553, each with a test
      that fails without its fix: task data was linked, which dsh's search does not follow; web search with another
      backend's model went to DeepSeek on DEEPSEEK_API_KEY; calls without reported tokens were priced at $0.00; the
      deployment marker could make a folder above the workspace the instructions' root; the deployed tool server did
      not fail the session when it could not start; the size check left out dsh's 304-byte frame; the AHE texts named
      Codex's stream for Pi and dsh; an effort level or permission mode dsh lacks failed every session instead of the
      config; docstrings (the deploy command named no profile); tests for an AgentEvolver step, the deploy's limit and
      a server name dsh refuses, and a recorded SIGTERM stream (stopped.jsonl). Explained, not changed: dsh injects
      a nested AGENTS.md when its file tools touch a folder (the templates' copies equal their context); the
      deployed skills row replaces a skill-filesystem config of the user's own (documented); the context window.
      The round-1 message overstated one change: a dsh usage error was already an error record per task, not a wrong
      answer; it now stops the run.
    • Round 3, on 9d87553: two reviewers, 16 findings and questions (two found by both). Fixed in 2021944,
      each with a test that fails without its fix. One was a regression of round 2's own fix: its shared pricing rule
      made Claude Code's results of refused requests (one turn, no tokens) cost unknown, and the AHE and RRSI merges
      then dropped a step's known spend from the budget; it is replaced by the dsh-only unreported_calls. The others:
      dsh's zstd transcripts escaped key scrubbing; the dsh learner was given an adapter's AGENTS.md as instructions;
      the dsh Evolver's settings were not checked at load, and YAML's bare off was not off; the search rule went by
      the backend's name; a missing context_window failed every request on vLLM without saying why; the task-data
      binding had no test CI runs; wording (the deployed server row, a run dsh never started, the size check's scope,
      the linked-data warning). Explained, not changed: dsh's refusal of every command without bubblewrap or Landlock
      (not reproduced: a container without privileges fell back to Landlock); a session with bound data stopped by
      SIGTERM ends without an event.
    • Round 4, on 2021944: two reviewers, 14 findings, questions and checklist items (two found by both). Fixed in f4b7b09,
      each with a test that fails without its fix: a step dsh retried after a refused request lost its cache tokens
      and was priced as known (dsh drops the step's cache counts; recorded: retried.jsonl), which also corrects round
      3's account of retries; the dsh Evolver's checks missed the method named with its params and refused an effort
      with the run's own model (they now run on the Trainer's params, in hermit check and before train's first
      step); efforts dsh's own provider refuses passed; evolver.dsh.instruction_files was dropped, not refused; the
      caller's DEEPSEEK_SEARCH_BASE_URL moved dsh's search without the sandbox; the context-window hint cut dsh's
      error. Explained, not changed: a home of compressed transcripts from earlier commits; an Evolver with no model
      reference (as Pi's on main); agent.dsh.instruction_files, like other free-form settings, does not reach the
      acting agent; a successful result keeps an unread failure text, as Pi's does.
    • Round 5, on f4b7b09: two reviewers, 14 findings, questions and test gaps (one found by both). Fixed in
      f1a92ea, each with a test that fails without its fix: round 4's unknown cost for a retried step made a whole
      session's spend invisible to a USD budget, while its tokens are known and hermit prices a cache write as input
      (now priced); the caller's DEEPSEEK_BASE_URL moved dsh's own provider without the sandbox; the Evolver's check
      refused its own values, and the acting agent's instruction_files: false, which would cut the adapter's context,
      was dropped; tests now pin the session's tool row (failOnStartupError, its 30-minute call timeout), every row
      DSH_OFF turns off and all six outage codes; wording: effort: off asks a bound model for no level (its
      server's default), and the context-window hint keeps dsh's error line as far as it fits. Explained, not changed:
      a Responses server's own total_tokens is taken as pi-ai gives it; a session with an unreported call has an
      unknown cost, which a USD budget does not count (as any unknown cost); a deployed tool server's stderr reaches
      dsh's (not reproduced as a problem).
    • Round 6, on f1a92ea: two reviewers, 11 findings (none found by both); the fixes change claims and tests,
      no behaviour. Fixed in 37aee8c: tests, each failing without what it checks: a session's own env reaching dsh
      had no test; the integration test of dsh's own sandbox now shows a command's write outside the workspace refused
      (it passed with the sandbox off); a bash call that failed to run has no exit code, and a session whose folder, or
      the folder it lies in, is a link is not resumed (neither was tested); and wording, in the reader's account: a
      stopped session's last step is not counted (its docstring said it was a call of unknown tokens), dsh's own
      adapter, not pi-ai,
      drops a cache count of 0, and a retried step's price is never less only for a price whose cached input is no
      dearer than its input, and on the Responses API only as far as the server's total_tokens holds the tokens; a
      stale test docstring. Explained, not changed: a sandbox's grant of DEEPSEEK_BASE_URL does not reach dsh (see
      Layer 2); a task run again after an outage keeps only the last attempt's usage, for every CLI (see Known dsh
      behaviours).
    • Round 7, on 37aee8c: a workflow of five reviewers, one lens each (usage and cost; settings and entry
      points; the session at run time; parity with Pi and shared code; tests and claims), each finding then reproduced
      by a separate agent told to refute it. 34 findings (27 distinct), all confirmed, none a wrong result in a correct
      configuration. One was rated medium: no test guarded which steps of dsh's stream count as calls, the rules a USD
      budget rests on; tested in 1bbd57c. One more, rated medium by its reviewer and low by the agent that
      reproduced it (the code was right), was changed at the author's request: no test held a command dsh moves to a
      background job at its timeout, what a timeout gives in a session unless dsh's job limit is reached, and only one
      check keeps it from reading as exit 0. f4792e0 records one (promoted.jsonl) and tests it, and also a
      command stopped from outside the call ([stopped: <reason>]), whose two results are written by hand as dsh's
      bash tool renders them: no headless session writes one. The Pi fixes this branch carried moved to Read a Pi acting agent's event stream as Pi's in parse_log and the AHE Evolver's texts #91 and mwm and RRSI: refuse a learner backend other than Codex or Claude Code at setup #92
      (207ea88), so that this PR changes only dsh, and the check under bubblewrap keeps its home under
      ~/.cache/hermit-tests. Corrected in this description: its claims that mwm and RRSI both ran pi without its
      model (RRSI ran it on its model, with role settings Pi does not read), that round 6's fixes were all tests, that
      a stopped session's cost is always known, that the acting agent's check at load is as Claude Code's effort check
      (that one runs at load only on a login), and its list of usage dsh does not report (web searches were missing).
      The other low ones are not changed (the author's call): see Known issues from review rounds 7 and 9.
    • Round 8, on f4792e0: two reviewers, one on the test and its recorded stream, one on the claims, each
      finding then reproduced by a separate agent told to refute it. 6 findings (5 distinct), all confirmed, one in
      part. Two were medium, both fixed in d25958a with recorded results: dsh prints the timeout it applied to a
      command, the model's when it gave one (1000.5), and both timeout markers were matched only at a whole number
      of milliseconds, so a command still running at such a timeout read as exit 0, and one killed at it that trapped
      the signal as its trapped exit; and the marker alone, what a command that printed nothing yet gets, was
      untested. The low ones are corrected: two in this round's own text (a docstring's order of dsh's markers, and
      that the [stopped: …] results are written by hand), and this description's diff size, stale since 1bbd57c
      (f4792e0 added a file).
    • Round 9, on d25958a: two reviewers, one on the code and its tests, one on the claims, each finding then
      reproduced by a separate agent told to refute it. 6 findings (4 distinct), all low, all confirmed. Three were
      wording this change's own text got wrong, corrected in a43698d and here: dsh prints the timeout it applied
      (60 s when the model gives none, at most 600 s), not always the model's; DSH_NO_OUTPUT's comment against a
      background marker that stands alone; round 8's account of its low findings. The fourth, the exponent form of a
      timeout untested, is not changed: see Known issues.
  • This description: read end to end against the code at 207ea88 and the run records it cites; the parts
    1bbd57c, f4792e0, d25958a and a43698d change, against those commits.
  • Independent review (post-PR audit): pending.

🤖 Generated with Claude Code

iamziyuzhao and others added 8 commits November 8, 2026 20:23
…Evolver)

DeepSeek Harness (`dsh`, npm @deepseek-ai/dsh 0.2.2-rc.1) beside codex, claude and pi, in the
same three layers as Pi (#89), on the runner's outcome() of #77.

- Layer 1: a <backend>/<model> reference becomes a route of dsh's pi-ai adapter
  (Binding.dsh_provider): the backend's URL in the first dialect it serves, the key by its
  variable's name (apiKeyEnv), the context window. dsh reports no cost: the backend's price
  applies.
- Layer 2: Dsh(CliAgent) runs `dsh --profile headless --json` in a private DSH_HOME, HOME
  and DSH_AGENTS_HOME, telemetry off, the caller's own DSH_* variables dropped. The
  session's settings are a patch layer of dsh's own rows (--patch): the bound model as the
  default model (effort as its reasoningEffort, image_input as Pi's), the call's tools as an
  MCP server offered directly (1800 s per call), web search and fetch off unless asked for,
  no title request, session log, plugin list, sub-agents or workflows. A server that needs
  no key gets the placeholder key through HERMIT_DSH_NO_KEY (a route that names none is
  refused by dsh). Approvals are never asked (--approval-policy never); permission_mode
  picks dsh's own sandbox. Resume by --session-id when the home keeps the session; the
  workspace's .dsh/skills are also the session's own (dsh reads project skills only at the
  nearest folder with a .git). outcome(): a run without a completed `final` has failed,
  whatever dsh's exit code (SIGTERM exits 0). Usage from step_end events; the trajectory
  from the stream, a command's exit code from dsh's end markers (a timeout or a stop:
  unknown; "(no output)": empty); every unfinished end is an error step. dsh's outage codes
  (RATE_LIMIT, SERVER, TIMEOUT, TRANSPORT, EMPTY_RESPONSE, STREAM_CLOSED) join UNAVAILABLE.
  Roles, a login and container isolation are refused.
- Layer 3: hermit/hosts/dsh.py (`hermit deploy --host dsh`): AGENTS.md, .dsh/skills and
  the adapter's tool server (120 s per call) as an insert row of .dsh/hermit.patch.yml,
  which the acting agent passes with --patch. The AgentEvolver, the loader and the config
  accept dsh; methods/dsh/ is the new Evolver (dsh's own sandbox off in the evolver
  sandbox, as Codex's is).
- Tests: tests/core/test_dsh_agent.py over streams recorded from dsh 0.2.2-rc.1
  (tests/fixtures/dsh) and, marked integration, the installed dsh: a session unsandboxed
  and in bubblewrap, the acting agent inside a git repository (the deployed skill offered,
  the adapter's own tool server answering), a keyless server, and --dump-config holding
  every row the session patches; the deploy host; selection entries.

Checked against dsh's source at the tag dsh-v0.2.2-rc.1 and with the installed dsh. First
built on #89's earlier head; rebuilt on main after #89 merged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each fix was checked against dsh's source at dsh-v0.2.2-rc.1 and, where it shows, against
the installed dsh (new recorded streams in tests/fixtures/dsh); each has a test that fails
without it.

- Stream reading: a bash result dsh's --json cut at 8 KiB (`truncated`) lost its exit
  marker and read as exit 0; now unknown, and the text says it was cut. A command started
  in the background has no status yet (unknown). The end markers are anchored at the very
  end (\Z), as dsh's JavaScript reads them, so a command that printed a marker of its own
  and exited 0 is a clean exit.
- Usage: a step that answered counts as a call even when the provider reported no usage
  (dsh reports zeros), and so does a step dsh reports no usage for (an attempt without
  any); what dsh never reports (compaction, a step's partial usage) is documented.
  parse_log reads Pi's and dsh's streams with their own readers.
- outcome(): a session whose tools did not start (dsh writes the MCP row's error on stderr
  and runs on without them) has failed. A run that wrote only an `error` event (a usage
  error) or nothing (a working directory's .env that sets PATH, PYTHONPATH, ...) started
  no session, so the acting agent reports it instead of recording each task as a wrong
  answer: outcome() may now say `started` (shared runner; absent, as before).
- The workspace is the project wherever it lies: the session's layer sets the
  skill-filesystem row to the workspace's .dsh/skills alone and the agent-instructions row
  to the deployment's marker (.hermit-deploy.json) as the project root, so a workspace
  inside a git repository gets its own skills and none of the repository's skills or
  AGENTS.md (they outranked the adapter's). This replaces copying the skills to the home.
- `--add-dir` takes the real path of an existing directory, as dsh requires (a link failed
  every command and write); an effort without a model is refused; user_home() follows
  DSH_HOME.
- The acting agent passes agent.dsh.permission_mode on (it was dropped) and refuses a mode
  dsh has not.
- hermit deploy --host dsh: the patch layer also names the workspace's .dsh/skills (a
  workspace below a repository's root kept none), and a context past the 65,536 bytes dsh
  reads of a project's instructions is refused.
- The AHE template has a `dsh` wire (as Pi's: hooks run by hand hand the memory over); mwm
  and RRSI refuse a backend other than Codex or Claude Code at setup (pi and dsh were
  accepted and ran without their model).
- Docstrings say what dsh reads (.env files, the instruction chain) and sends (its own
  provider's ids).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each fix was checked against dsh's source at dsh-v0.2.2-rc.1 and, where it shows, against the
installed dsh; each has a test that fails without it.

- A task's data is bound in the workspace, not linked (binds_mounts, as for Claude Code): dsh's
  glob and grep run ripgrep without --follow, so a linked folder was never listed or matched.
  dsh's own sandbox still runs nested in that mount namespace (integration test).
- Web search with a model of another backend: dsh's search is DeepSeek's own service (its
  Messages API on DEEPSEEK_API_KEY), so the session searched on a provider and key the run did
  not bind, outside its usage. Such a session gets web fetch only, as Codex's hosted search is
  off for a backend other than OpenAI's; a DeepSeek model or dsh's own provider keeps search.
- Usage: a step dsh reports no usage for counts as a call only if it produced something (a step
  whose requests all failed is none), and price_usage leaves calls of which no token was
  reported unpriced: their cost is unknown, not $0.00 (shared: a usage with calls and no tokens
  is no longer priced at 0 for any CLI).
- The instructions' project root: no root marker, so it is the working directory, the workspace.
  The deployment marker (.hermit-deploy.json, which every host's deployment writes) made a
  session without a deployment take a deployed folder above it for its root, and read its
  AGENTS.md.
- hermit deploy --host dsh: the tool server's row fails at a session's start when dsh cannot
  start it (failOnStartupError, as Codex's `required`), which the acting agent reads as a failed
  session; it was retried in the background while the session ran without the adapter's tools.
  The 65,536-byte check counts dsh's frame around AGENTS.md (304 bytes, a </system-reminder>
  one byte more): an AGENTS.md just under the cap was deployed and cut. Docstring: the command
  names its profile; the skills row replaces a skill-filesystem config of the user's own.
- dsh's effort levels and permission modes are checked as the run config loads (an unknown one
  failed every session, one task at a time), and an effort level dsh has not is refused in the
  session.
- The AHE learners' texts name a Pi or dsh acting agent's stream (agent-pi.jsonl,
  agent-dsh.jsonl) instead of agent-codex.jsonl (Pi's entry was missing on main too).
- Tests: a recorded stream of a session stopped by SIGTERM (stopped.jsonl: exit 0, no `final`,
  an `error` event) replaces a cut success stream, and is run through the acting agent and an
  AgentEvolver step; the deploy refuses at the exact limit and an MCP server name dsh refuses.

Correction to round 1's message: on 03fafff a dsh usage error was already an error record per
task (failed), not a wrong answer; `started` now makes the acting agent stop the run and say so.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each fix was checked against dsh's source at dsh-v0.2.2-rc.1 and, where it shows, against the
installed dsh; each has a test that fails without it.

- Usage: round 2's shared rule (a usage with calls and no tokens is not priced) is gone. It
  made Claude Code's results of refused requests (a 529, a 401: one turn, no tokens) cost
  unknown instead of $0.00, and the AHE and RRSI evolvers' merged usage then dropped the
  step's known spend from the budget. Usage gains `unreported_calls`, the calls whose tokens a
  CLI did not report; only dsh's reader sets it (a step that produced something without
  usage: the provider reported none, or dsh retried the step and so left its usage out), and
  price_usage leaves such a usage's cost unknown rather than pricing the part reported, which
  under rate limits may be far less.
- dsh's transcripts are plain JSON lines (the session layer sets session-persistence-jsonl to
  compression none, its root as dsh's own `!!js dshHomePath('sessions')`): they were zstd,
  which hermit.redact cannot scrub a key from. A scrubbed transcript still resumes.
- The dsh Evolver's sessions take no instruction file (instruction_files false: the
  agent-instructions budget 0): dsh adds a folder's AGENTS.md when its file tools touch it, so
  the learner was given the adapter's AGENTS.md, the acting agent's policy, as instructions.
  Codex's learner is given none.
- dsh's settings are checked as the run loads for the dsh Evolver too (an effort it has not
  failed every learner step, after its rollouts); evolver.dsh.permission_mode is refused (the
  Evolver runs dsh's own sandbox off), and YAML's bare `off`, a boolean, is `off`.
- Web search is kept only for a model reached at api.deepseek.com on DEEPSEEK_API_KEY, where
  dsh's search goes (a `deepseek` backend moved by DEEPSEEK_BASE_URL searched elsewhere).
- A run that ends CONTEXT_WINDOW_EXCEEDED with a backend that declares no context_window says
  to declare it (recorded: context_window.jsonl, vLLM's refusal of dsh's 128,000 output
  tokens; with the window declared dsh sizes the request and completes).
- A unit test with the fake dsh for the task data's binding, which CI runs where bubblewrap can
  bind (the only test needed the dsh CLI).
- Wording: the deployed tool server's row makes dsh report a server it cannot start and the
  acting agent fail the session (dsh itself goes on); a run dsh never started is each task's
  error record, not a stopped run (round 2's message said so); the deploy's size check is exact
  for a session at the workspace's root; the linked-data warning names dsh's glob and grep.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each fix was checked against dsh's source at dsh-v0.2.2-rc.1 and against the installed dsh;
each has a test that fails without it.

- A step dsh retried after the provider refused a request lost its cache counts: the refused
  attempt reports zeros, pi-ai reports a count of 0 as none, and dsh sums a count into the
  step's only when every attempt reported it, while the step's totalTokens still holds them.
  The reader counts what the total holds beyond the counts reported as input, and the call as
  unreported (its cost unknown: those tokens were read from the cache or written to it, which
  are priced apart). Recorded: retried.jsonl (a 503, then 1000 prompt tokens, 900 cached:
  dsh reports inputTokens 100, totalTokens 1010). Round 3 said such a step's usage is left
  out; dsh does that only for an attempt that reports no usage at all (a stream that ended
  before its last event, an attempt a stop cut short).
- The dsh Evolver's settings are checked from its params as the Trainer has them (its method's
  own under the run's, the model bound), in login.evolver_problems, which train runs before its
  first step and hermit check now reports for an Evolver without a login. Round 3's check, as
  the run loaded, missed the dsh method named with its params (`backend: dsh` comes from the
  method's manifest) and refused an effort given with the run's own model. The acting agent's
  are still checked as the run loads; one function (cli.dsh_problems) holds both, and
  isolate.py's change is that call.
- dsh's own provider (a model of no binding, DeepSeek's API) takes the efforts off, low, high
  and max; another level failed every session (UNSUPPORTED_REASONING_EFFORT).
- evolver.dsh.instruction_files is refused, as evolver.dsh.permission_mode is: the Evolver
  dropped it.
- The caller's DEEPSEEK_SEARCH_BASE_URL is dropped with its DSH_* variables: without the
  sandbox it moved dsh's web search off DeepSeek's API.
- The context_window hint follows dsh's error whole (once), not its first 300 characters.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each fix was checked against dsh's source at dsh-v0.2.2-rc.1 and against the installed dsh;
each has a test that fails without it.

- A step dsh retried after a refused request is priced again, its dropped cache tokens counted
  as input. Round 4 made such a session's cost unknown, and a USD budget then counted none of
  its spend (rollouts and learner steps add no dollars for an unknown cost), under the very load
  that causes retries. Its tokens are known, and hermit prices a cache write as input: the price
  is the step's with the counts reported, or more if those tokens were read from the cache,
  never less. A call stays unreported, and its session's cost unknown, only when dsh reported
  none of its tokens.
- The caller's DEEPSEEK_BASE_URL is dropped as DEEPSEEK_SEARCH_BASE_URL is: without the sandbox
  it sent the requests of dsh's own provider, with DEEPSEEK_API_KEY, elsewhere. A variable the
  session is given (`env`) still passes.
- dsh's settings check takes what each agent uses: the Evolver's danger-full-access and
  instruction_files false are accepted (round 4 refused them); the acting agent's
  instruction_files false is refused (it was dropped; it would cut the adapter's context, which
  reaches dsh as its AGENTS.md).
- Tests pin the session's tool row (failOnStartupError, the 30-minute call timeout), every row
  DSH_OFF turns off, all six outage codes, and dsh's own provider's levels for the Evolver; the
  outage test is named for what it checks.
- Wording: effort off asks a bound model for no reasoning level, which leaves its server's
  default (dsh's own provider turns thinking off); the context-window hint keeps dsh's error
  line as far as it fits.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each finding was checked against dsh's source at dsh-v0.2.2-rc.1 and the installed dsh; each
test added fails without what it checks. No behaviour changes: the fixes are tests and claims.

- Tests: a session's own `env` reaches dsh while the caller's DEEPSEEK_BASE_URL is dropped; the
  integration test of dsh's own sandbox shows a command's write outside the workspace refused
  (it passed with the sandbox off); a bash call that failed to run (a result dsh marks `error`)
  has no exit code; a linked session folder, or a session in a linked folder, is not resumed;
  a stopped session's usage is pinned (none).
- The reader's account: the last step of a session a stop cut short is not counted (dsh writes
  a step's usage at its end, after its tool calls ran, and a stopped session writes nothing
  more); the docstring said such a step was a call of unknown tokens. dsh's own adapter, not
  pi-ai, drops a cache count of 0. A retried step's price is never less than with its counts
  for a price whose cached input is no dearer than its input; on the Responses API the total
  is the server's own `total_tokens`, taken as given.
- dropped_env says why a sandbox's grant of DEEPSEEK_BASE_URL does not reach dsh (hermit's own
  is its deepseek backend's chat completions root, which dsh reads as its own provider's
  Messages root) and that `env` gives dsh its own; the Settings paragraph says so too.
- A stale test docstring: the Evolver's own permission mode and instruction_files are taken.

Not changed, disclosed: the acting agent's re-run after an outage keeps only its last
attempt's usage, for every CLI (shared code, a separate change).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Pi fixes this branch carried change Pi's behaviour on main, not dsh's: parse_log's Pi
reader, the AHE Evolver's texts for a Pi acting agent's stream, and mwm and RRSI refusing a Pi
learner. They go to their own PRs (fix/pi-stream-readers, fix/mwm-rrsi-backend-check). This
branch keeps dsh's entries only: parse_log reads a dsh stream as dsh's, the AHE texts name a
dsh acting agent's stream, and mwm and RRSI refuse dsh (the other PR's general check covers it
too, whichever merges first).

The integration test of dsh's own sandbox keeps the session's home under
~/.cache/hermit-tests, as the test checklist asks of a check under bubblewrap (it was under
/var/tmp).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review round 7 (five reviewers on 37aee8c, one lens each, every finding then reproduced by a
separate agent told to refute it) found 34 issues, all confirmed: one of medium severity, the
rest low. The medium one: no test guarded the reader's three rules for a step whose tokens dsh
did not report (a step that only called a tool is a call, so is one that only thought, and what
a step produced does not carry over to the next); each, undone, left every test passing while
it misstated a session's cost. The new test fails under each. The low ones are listed in the
PR description, not changed.

Also: a round-6 test docstring rewrapped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
iamziyuzhao and others added 3 commits November 9, 2026 13:21
In a hermit dsh session a command still running at its timeout (dsh's default is 60 s) is
moved to a background job, unless dsh's job limit is reached: the session's layer leaves
tool-bash's promoteOnTimeout at its default, true. Its result ends `[still running after
<ms>ms; moved to background job <id>]` with no exit marker, and only the DSH_BACKGROUND check
keeps it from reading as exit 0. No test held that case: bash.jsonl was recorded with
promotion off. Review round 7 found this (rated medium by its reviewer, low by the agent that
reproduced it, as the code was right); the author asked for it fixed.

promoted.jsonl is recorded through make("dsh").run, as the other Base Agent streams are, and
the test now reads it, and also a command stopped from outside the call (`[stopped: <reason>]`,
which no headless session writes, so written as dsh's tool-bash renders it), with and without
a marker after it. Each check undone (the background check, the `stopped:` alternative, the
markers after a stop) fails the test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review round 8 (two reviewers on f4792e0, one on the test and its fixture, one on the claims,
each finding then reproduced by a separate agent told to refute it) found:

- dsh prints a command's timeout as the model gave it, a JavaScript number (`1000.5`, `0.5`),
  and both timeout markers were matched only at a whole number of milliseconds: a command still
  running at such a timeout, moved to a background job, read as exit 0, and one killed at it that
  trapped the signal read as its trapped exit (rated medium). Both markers now match any number
  dsh prints. Recorded: promoted.jsonl's third call (1000.5 ms, moved to the background) and
  bash.jsonl's ninth (1000.5 ms, killed, exit 0 trapped).
- The test of a promoted command held only one with output before its marker; the marker alone,
  what a command that printed nothing yet gets, rests on DSH_BACKGROUND matching without a newline
  before it, which nothing held (rated medium). promoted.jsonl's second call is `sleep 5`.
- Wording (low, in f4792e0's own docstring): the stop marker comes before the signal or exit
  marker, not before every other marker (a sandbox denial's come first).

Each fix undone fails its test: the marker with a newline required before it, and either marker
at whole milliseconds only. bash.jsonl is recorded again by the same script, its first eight
results unchanged; its steps' count in the usage test follows (10).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review round 9 (two reviewers on d25958a, one on the code and its tests, one on the claims, each
finding then reproduced by a separate agent told to refute it) found 6 issues (4 distinct), all
low. Three are wording this change's own text got wrong, corrected here (comments, a docstring
and the fixture README; no code or test changes):

- dsh prints the timeout it applied, not always the model's: its default (60 s) when the model
  gives none, and at most its cap (600 s).
- DSH_NO_OUTPUT's comment said `(no output)` comes before the marker of a command that printed
  nothing, which a command moved to the background does not get (its marker stands alone).

The fourth, the exponent form both timeout patterns match (`1e-7`, a timeout under 0.000001 ms)
held by no test, is listed in the PR description, not changed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant