Repository navigation
Add DeepSeek Harness (dsh) as a CLI Base Agent (acting agent and dsh Evolver) - #90
Draft
iamziyuzhao wants to merge 12 commits into
Draft
iamziyuzhao wants to merge 12 commits into
iamziyuzhao wants to merge 12 commits into
Conversation
…Evolver) DeepSeek Harness (`dsh`, npm @deepseek-ai/dsh 0.2.2-rc.1) beside codex, claude and pi, in the same three layers as Pi (#89), on the runner's outcome() of #77. - Layer 1: a <backend>/<model> reference becomes a route of dsh's pi-ai adapter (Binding.dsh_provider): the backend's URL in the first dialect it serves, the key by its variable's name (apiKeyEnv), the context window. dsh reports no cost: the backend's price applies. - Layer 2: Dsh(CliAgent) runs `dsh --profile headless --json` in a private DSH_HOME, HOME and DSH_AGENTS_HOME, telemetry off, the caller's own DSH_* variables dropped. The session's settings are a patch layer of dsh's own rows (--patch): the bound model as the default model (effort as its reasoningEffort, image_input as Pi's), the call's tools as an MCP server offered directly (1800 s per call), web search and fetch off unless asked for, no title request, session log, plugin list, sub-agents or workflows. A server that needs no key gets the placeholder key through HERMIT_DSH_NO_KEY (a route that names none is refused by dsh). Approvals are never asked (--approval-policy never); permission_mode picks dsh's own sandbox. Resume by --session-id when the home keeps the session; the workspace's .dsh/skills are also the session's own (dsh reads project skills only at the nearest folder with a .git). outcome(): a run without a completed `final` has failed, whatever dsh's exit code (SIGTERM exits 0). Usage from step_end events; the trajectory from the stream, a command's exit code from dsh's end markers (a timeout or a stop: unknown; "(no output)": empty); every unfinished end is an error step. dsh's outage codes (RATE_LIMIT, SERVER, TIMEOUT, TRANSPORT, EMPTY_RESPONSE, STREAM_CLOSED) join UNAVAILABLE. Roles, a login and container isolation are refused. - Layer 3: hermit/hosts/dsh.py (`hermit deploy --host dsh`): AGENTS.md, .dsh/skills and the adapter's tool server (120 s per call) as an insert row of .dsh/hermit.patch.yml, which the acting agent passes with --patch. The AgentEvolver, the loader and the config accept dsh; methods/dsh/ is the new Evolver (dsh's own sandbox off in the evolver sandbox, as Codex's is). - Tests: tests/core/test_dsh_agent.py over streams recorded from dsh 0.2.2-rc.1 (tests/fixtures/dsh) and, marked integration, the installed dsh: a session unsandboxed and in bubblewrap, the acting agent inside a git repository (the deployed skill offered, the adapter's own tool server answering), a keyless server, and --dump-config holding every row the session patches; the deploy host; selection entries. Checked against dsh's source at the tag dsh-v0.2.2-rc.1 and with the installed dsh. First built on #89's earlier head; rebuilt on main after #89 merged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each fix was checked against dsh's source at dsh-v0.2.2-rc.1 and, where it shows, against the installed dsh (new recorded streams in tests/fixtures/dsh); each has a test that fails without it. - Stream reading: a bash result dsh's --json cut at 8 KiB (`truncated`) lost its exit marker and read as exit 0; now unknown, and the text says it was cut. A command started in the background has no status yet (unknown). The end markers are anchored at the very end (\Z), as dsh's JavaScript reads them, so a command that printed a marker of its own and exited 0 is a clean exit. - Usage: a step that answered counts as a call even when the provider reported no usage (dsh reports zeros), and so does a step dsh reports no usage for (an attempt without any); what dsh never reports (compaction, a step's partial usage) is documented. parse_log reads Pi's and dsh's streams with their own readers. - outcome(): a session whose tools did not start (dsh writes the MCP row's error on stderr and runs on without them) has failed. A run that wrote only an `error` event (a usage error) or nothing (a working directory's .env that sets PATH, PYTHONPATH, ...) started no session, so the acting agent reports it instead of recording each task as a wrong answer: outcome() may now say `started` (shared runner; absent, as before). - The workspace is the project wherever it lies: the session's layer sets the skill-filesystem row to the workspace's .dsh/skills alone and the agent-instructions row to the deployment's marker (.hermit-deploy.json) as the project root, so a workspace inside a git repository gets its own skills and none of the repository's skills or AGENTS.md (they outranked the adapter's). This replaces copying the skills to the home. - `--add-dir` takes the real path of an existing directory, as dsh requires (a link failed every command and write); an effort without a model is refused; user_home() follows DSH_HOME. - The acting agent passes agent.dsh.permission_mode on (it was dropped) and refuses a mode dsh has not. - hermit deploy --host dsh: the patch layer also names the workspace's .dsh/skills (a workspace below a repository's root kept none), and a context past the 65,536 bytes dsh reads of a project's instructions is refused. - The AHE template has a `dsh` wire (as Pi's: hooks run by hand hand the memory over); mwm and RRSI refuse a backend other than Codex or Claude Code at setup (pi and dsh were accepted and ran without their model). - Docstrings say what dsh reads (.env files, the instruction chain) and sends (its own provider's ids). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each fix was checked against dsh's source at dsh-v0.2.2-rc.1 and, where it shows, against the installed dsh; each has a test that fails without it. - A task's data is bound in the workspace, not linked (binds_mounts, as for Claude Code): dsh's glob and grep run ripgrep without --follow, so a linked folder was never listed or matched. dsh's own sandbox still runs nested in that mount namespace (integration test). - Web search with a model of another backend: dsh's search is DeepSeek's own service (its Messages API on DEEPSEEK_API_KEY), so the session searched on a provider and key the run did not bind, outside its usage. Such a session gets web fetch only, as Codex's hosted search is off for a backend other than OpenAI's; a DeepSeek model or dsh's own provider keeps search. - Usage: a step dsh reports no usage for counts as a call only if it produced something (a step whose requests all failed is none), and price_usage leaves calls of which no token was reported unpriced: their cost is unknown, not $0.00 (shared: a usage with calls and no tokens is no longer priced at 0 for any CLI). - The instructions' project root: no root marker, so it is the working directory, the workspace. The deployment marker (.hermit-deploy.json, which every host's deployment writes) made a session without a deployment take a deployed folder above it for its root, and read its AGENTS.md. - hermit deploy --host dsh: the tool server's row fails at a session's start when dsh cannot start it (failOnStartupError, as Codex's `required`), which the acting agent reads as a failed session; it was retried in the background while the session ran without the adapter's tools. The 65,536-byte check counts dsh's frame around AGENTS.md (304 bytes, a </system-reminder> one byte more): an AGENTS.md just under the cap was deployed and cut. Docstring: the command names its profile; the skills row replaces a skill-filesystem config of the user's own. - dsh's effort levels and permission modes are checked as the run config loads (an unknown one failed every session, one task at a time), and an effort level dsh has not is refused in the session. - The AHE learners' texts name a Pi or dsh acting agent's stream (agent-pi.jsonl, agent-dsh.jsonl) instead of agent-codex.jsonl (Pi's entry was missing on main too). - Tests: a recorded stream of a session stopped by SIGTERM (stopped.jsonl: exit 0, no `final`, an `error` event) replaces a cut success stream, and is run through the acting agent and an AgentEvolver step; the deploy refuses at the exact limit and an MCP server name dsh refuses. Correction to round 1's message: on 03fafff a dsh usage error was already an error record per task (failed), not a wrong answer; `started` now makes the acting agent stop the run and say so. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each fix was checked against dsh's source at dsh-v0.2.2-rc.1 and, where it shows, against the
installed dsh; each has a test that fails without it.
- Usage: round 2's shared rule (a usage with calls and no tokens is not priced) is gone. It
made Claude Code's results of refused requests (a 529, a 401: one turn, no tokens) cost
unknown instead of $0.00, and the AHE and RRSI evolvers' merged usage then dropped the
step's known spend from the budget. Usage gains `unreported_calls`, the calls whose tokens a
CLI did not report; only dsh's reader sets it (a step that produced something without
usage: the provider reported none, or dsh retried the step and so left its usage out), and
price_usage leaves such a usage's cost unknown rather than pricing the part reported, which
under rate limits may be far less.
- dsh's transcripts are plain JSON lines (the session layer sets session-persistence-jsonl to
compression none, its root as dsh's own `!!js dshHomePath('sessions')`): they were zstd,
which hermit.redact cannot scrub a key from. A scrubbed transcript still resumes.
- The dsh Evolver's sessions take no instruction file (instruction_files false: the
agent-instructions budget 0): dsh adds a folder's AGENTS.md when its file tools touch it, so
the learner was given the adapter's AGENTS.md, the acting agent's policy, as instructions.
Codex's learner is given none.
- dsh's settings are checked as the run loads for the dsh Evolver too (an effort it has not
failed every learner step, after its rollouts); evolver.dsh.permission_mode is refused (the
Evolver runs dsh's own sandbox off), and YAML's bare `off`, a boolean, is `off`.
- Web search is kept only for a model reached at api.deepseek.com on DEEPSEEK_API_KEY, where
dsh's search goes (a `deepseek` backend moved by DEEPSEEK_BASE_URL searched elsewhere).
- A run that ends CONTEXT_WINDOW_EXCEEDED with a backend that declares no context_window says
to declare it (recorded: context_window.jsonl, vLLM's refusal of dsh's 128,000 output
tokens; with the window declared dsh sizes the request and completes).
- A unit test with the fake dsh for the task data's binding, which CI runs where bubblewrap can
bind (the only test needed the dsh CLI).
- Wording: the deployed tool server's row makes dsh report a server it cannot start and the
acting agent fail the session (dsh itself goes on); a run dsh never started is each task's
error record, not a stopped run (round 2's message said so); the deploy's size check is exact
for a session at the workspace's root; the linked-data warning names dsh's glob and grep.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each fix was checked against dsh's source at dsh-v0.2.2-rc.1 and against the installed dsh; each has a test that fails without it. - A step dsh retried after the provider refused a request lost its cache counts: the refused attempt reports zeros, pi-ai reports a count of 0 as none, and dsh sums a count into the step's only when every attempt reported it, while the step's totalTokens still holds them. The reader counts what the total holds beyond the counts reported as input, and the call as unreported (its cost unknown: those tokens were read from the cache or written to it, which are priced apart). Recorded: retried.jsonl (a 503, then 1000 prompt tokens, 900 cached: dsh reports inputTokens 100, totalTokens 1010). Round 3 said such a step's usage is left out; dsh does that only for an attempt that reports no usage at all (a stream that ended before its last event, an attempt a stop cut short). - The dsh Evolver's settings are checked from its params as the Trainer has them (its method's own under the run's, the model bound), in login.evolver_problems, which train runs before its first step and hermit check now reports for an Evolver without a login. Round 3's check, as the run loaded, missed the dsh method named with its params (`backend: dsh` comes from the method's manifest) and refused an effort given with the run's own model. The acting agent's are still checked as the run loads; one function (cli.dsh_problems) holds both, and isolate.py's change is that call. - dsh's own provider (a model of no binding, DeepSeek's API) takes the efforts off, low, high and max; another level failed every session (UNSUPPORTED_REASONING_EFFORT). - evolver.dsh.instruction_files is refused, as evolver.dsh.permission_mode is: the Evolver dropped it. - The caller's DEEPSEEK_SEARCH_BASE_URL is dropped with its DSH_* variables: without the sandbox it moved dsh's web search off DeepSeek's API. - The context_window hint follows dsh's error whole (once), not its first 300 characters. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each fix was checked against dsh's source at dsh-v0.2.2-rc.1 and against the installed dsh; each has a test that fails without it. - A step dsh retried after a refused request is priced again, its dropped cache tokens counted as input. Round 4 made such a session's cost unknown, and a USD budget then counted none of its spend (rollouts and learner steps add no dollars for an unknown cost), under the very load that causes retries. Its tokens are known, and hermit prices a cache write as input: the price is the step's with the counts reported, or more if those tokens were read from the cache, never less. A call stays unreported, and its session's cost unknown, only when dsh reported none of its tokens. - The caller's DEEPSEEK_BASE_URL is dropped as DEEPSEEK_SEARCH_BASE_URL is: without the sandbox it sent the requests of dsh's own provider, with DEEPSEEK_API_KEY, elsewhere. A variable the session is given (`env`) still passes. - dsh's settings check takes what each agent uses: the Evolver's danger-full-access and instruction_files false are accepted (round 4 refused them); the acting agent's instruction_files false is refused (it was dropped; it would cut the adapter's context, which reaches dsh as its AGENTS.md). - Tests pin the session's tool row (failOnStartupError, the 30-minute call timeout), every row DSH_OFF turns off, all six outage codes, and dsh's own provider's levels for the Evolver; the outage test is named for what it checks. - Wording: effort off asks a bound model for no reasoning level, which leaves its server's default (dsh's own provider turns thinking off); the context-window hint keeps dsh's error line as far as it fits. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each finding was checked against dsh's source at dsh-v0.2.2-rc.1 and the installed dsh; each test added fails without what it checks. No behaviour changes: the fixes are tests and claims. - Tests: a session's own `env` reaches dsh while the caller's DEEPSEEK_BASE_URL is dropped; the integration test of dsh's own sandbox shows a command's write outside the workspace refused (it passed with the sandbox off); a bash call that failed to run (a result dsh marks `error`) has no exit code; a linked session folder, or a session in a linked folder, is not resumed; a stopped session's usage is pinned (none). - The reader's account: the last step of a session a stop cut short is not counted (dsh writes a step's usage at its end, after its tool calls ran, and a stopped session writes nothing more); the docstring said such a step was a call of unknown tokens. dsh's own adapter, not pi-ai, drops a cache count of 0. A retried step's price is never less than with its counts for a price whose cached input is no dearer than its input; on the Responses API the total is the server's own `total_tokens`, taken as given. - dropped_env says why a sandbox's grant of DEEPSEEK_BASE_URL does not reach dsh (hermit's own is its deepseek backend's chat completions root, which dsh reads as its own provider's Messages root) and that `env` gives dsh its own; the Settings paragraph says so too. - A stale test docstring: the Evolver's own permission mode and instruction_files are taken. Not changed, disclosed: the acting agent's re-run after an outage keeps only its last attempt's usage, for every CLI (shared code, a separate change). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Pi fixes this branch carried change Pi's behaviour on main, not dsh's: parse_log's Pi reader, the AHE Evolver's texts for a Pi acting agent's stream, and mwm and RRSI refusing a Pi learner. They go to their own PRs (fix/pi-stream-readers, fix/mwm-rrsi-backend-check). This branch keeps dsh's entries only: parse_log reads a dsh stream as dsh's, the AHE texts name a dsh acting agent's stream, and mwm and RRSI refuse dsh (the other PR's general check covers it too, whichever merges first). The integration test of dsh's own sandbox keeps the session's home under ~/.cache/hermit-tests, as the test checklist asks of a check under bubblewrap (it was under /var/tmp). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Nov 9, 2026
Review round 7 (five reviewers on 37aee8c, one lens each, every finding then reproduced by a separate agent told to refute it) found 34 issues, all confirmed: one of medium severity, the rest low. The medium one: no test guarded the reader's three rules for a step whose tokens dsh did not report (a step that only called a tool is a call, so is one that only thought, and what a step produced does not carry over to the next); each, undone, left every test passing while it misstated a session's cost. The new test fails under each. The low ones are listed in the PR description, not changed. Also: a round-6 test docstring rewrapped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
In a hermit dsh session a command still running at its timeout (dsh's default is 60 s) is
moved to a background job, unless dsh's job limit is reached: the session's layer leaves
tool-bash's promoteOnTimeout at its default, true. Its result ends `[still running after
<ms>ms; moved to background job <id>]` with no exit marker, and only the DSH_BACKGROUND check
keeps it from reading as exit 0. No test held that case: bash.jsonl was recorded with
promotion off. Review round 7 found this (rated medium by its reviewer, low by the agent that
reproduced it, as the code was right); the author asked for it fixed.
promoted.jsonl is recorded through make("dsh").run, as the other Base Agent streams are, and
the test now reads it, and also a command stopped from outside the call (`[stopped: <reason>]`,
which no headless session writes, so written as dsh's tool-bash renders it), with and without
a marker after it. Each check undone (the background check, the `stopped:` alternative, the
markers after a stop) fails the test.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review round 8 (two reviewers on f4792e0, one on the test and its fixture, one on the claims, each finding then reproduced by a separate agent told to refute it) found: - dsh prints a command's timeout as the model gave it, a JavaScript number (`1000.5`, `0.5`), and both timeout markers were matched only at a whole number of milliseconds: a command still running at such a timeout, moved to a background job, read as exit 0, and one killed at it that trapped the signal read as its trapped exit (rated medium). Both markers now match any number dsh prints. Recorded: promoted.jsonl's third call (1000.5 ms, moved to the background) and bash.jsonl's ninth (1000.5 ms, killed, exit 0 trapped). - The test of a promoted command held only one with output before its marker; the marker alone, what a command that printed nothing yet gets, rests on DSH_BACKGROUND matching without a newline before it, which nothing held (rated medium). promoted.jsonl's second call is `sleep 5`. - Wording (low, in f4792e0's own docstring): the stop marker comes before the signal or exit marker, not before every other marker (a sandbox denial's come first). Each fix undone fails its test: the marker with a newline required before it, and either marker at whole milliseconds only. bash.jsonl is recorded again by the same script, its first eight results unchanged; its steps' count in the usage test follows (10). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review round 9 (two reviewers on d25958a, one on the code and its tests, one on the claims, each finding then reproduced by a separate agent told to refute it) found 6 issues (4 distinct), all low. Three are wording this change's own text got wrong, corrected here (comments, a docstring and the fixture README; no code or test changes): - dsh prints the timeout it applied, not always the model's: its default (60 s) when the model gives none, and at most its cap (600 s). - DSH_NO_OUTPUT's comment said `(no output)` comes before the marker of a command that printed nothing, which a command moved to the background does not get (its marker stands alone). The fourth, the exponent form both timeout patterns match (`1e-7`, a timeout under 0.000001 ms) held by no test, is listed in the PR description, not changed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Change
Adds DeepSeek Harness (
dsh; npm@deepseek-ai/dsh0.2.2-rc.1,deepseek-ai/deepseek-harness) as a fourth CLI Base Agent beside
codex,claudeandpi. It is oneCliAgentsubclass that reports its runs through the runner'soutcome()(#77). It can act (
agent.base: dsh) and learn (the newdshEvolver). It follows #89 (Pi).Based on
main(d5f0a64, after #89 merged). The first version of this PR was stacked on #89's earlier head. Thebranch was rebuilt on
main, so its diff is now only dsh: 50 files, +2394 −100. Draft: the documentation is notwritten yet (see Docs).
<backend>/<model>reference becomes a route of dsh's pi-ai adapter (Binding.dsh_provider): thebackend's URL in the first dialect it serves (as for Pi: chat completions, then Messages, then Responses), the key
by its variable's name (
apiKeyEnv, never the value) and the context window. dsh reports no cost, so the backend'sprice is applied to the usage it reports.
Dsh(CliAgent)runsdsh --profile headless --jsonin a private home (DSH_HOME,HOME,DSH_AGENTS_HOME), with telemetry off and the caller's ownDSH_*,DEEPSEEK_BASE_URLandDEEPSEEK_SEARCH_BASE_URLdropped (without the sandbox they moved dsh's own provider and its web search), whatevera sandbox lets through: hermit's own
DEEPSEEK_BASE_URLis itsdeepseekbackend's chat completions root, whichdsh reads as its own provider's Messages root. The session's
envgives dsh one of its own.--patch, written in the home):effortas itsreasoningEffort;image_input: truedeclares a modelthat takes images);
mcp__<server>__<tool>;web_search: true. dsh's search is DeepSeek's own service (its Messages APIon
DEEPSEEK_API_KEY), so unless the model is reached there on that key the session gets web fetch only, asCodex's hosted search is off for a backend other than OpenAI's;
hermit.redactscrubs akey a session wrote from its home, as from every other record;
DSH_TELEMETRY_DISABLED=1), the session log andplugin list that dsh adds to requests to DeepSeek's own API, and sub-agents and workflows (their tokens are in
no usage figure dsh reports).
from the workspace's
.dsh/skills(skill-filesystem), and instructions only from the working directory, theworkspace (
agent-instructionswith no root marker). Without them, a workspace inside a git repository also gotthat repository's skills and
AGENTS.md, which outranked the adapter's. The dsh Evolver's sessions take noinstruction file at all (
instruction_files: false): dsh adds a folder'sAGENTS.mdwhen its file tools touchthe folder, and an adapter's
AGENTS.mdis the acting agent's policy, which Codex's learner is never given.Code (
binds_mounts): dsh's glob and grep run ripgrep without--follow, so a link was never searched.localthrough a variable of the session(
HERMIT_DSH_NO_KEY), because dsh refuses every request of a route that names no key.agent.dsh.permission_mode, defaultworkspace-write).writablepaths are granted by their real paths, as dsh requires. Approvals are never asked(
--approval-policy never). A permission mode or an effort level dsh does not have is refused before anysession: the acting agent's as the run config loads, the dsh Evolver's by
hermit checkand beforetrain'sfirst step, from its params as the Trainer has them (the method's own and the run's model in). dsh's own provider
takes fewer levels (
off,low,high,max) than a bound model's route. YAML's bareoff, a boolean, isoff.--session-idwhen the home keeps the session (dsh resumes only in the directory the session ranin, which an AgentEvolver's calls share).
{{...}}otherthan
{{provider}}and{{model}}, so system instructions go at the top of the prompt as a marked block.login, and container isolation.outcome():finalevent has failed, whatever dsh's exit code (dsh exits 0 when SIGTERM stopsit).
sessionevent (a usage error, or a working directory's.envthat sets a name dshkeeps, such as
PATH) started no session (started: false), so the acting agent reports it instead ofrecording a wrong answer for every task.
statusstep_endevents (records.parse_dsh), each the sum of the step's attempts. Astep that produced something but whose tokens dsh did not report (the provider reported no usage, or an attempt
reported none and dsh left the step's usage out) is an unreported call, and the session's cost is then unknown
rather than the price of the tokens reported. A step dsh retried after a refused request loses its cache counts
(dsh keeps a count only when every attempt reported one), which its
totalTokensstill holds: those tokens arecounted as input and priced as hermit prices a cache write, never less than the step's price with the counts
(for a price whose cached input is no dearer than its input), so a USD budget still counts the session. On the
Responses API the total is the server's own
total_tokens, taken as given.CONTEXT_WINDOW_EXCEEDEDwith a backend that declares nocontext_windowsays to declare it. dsh sizes a model it does not know as 1,000,000 tokens with 128,000 output tokens a request,
which a vLLM server refuses; with the window declared, dsh sizes each request by it.
trajectory.dsh_steps). A command's exit code is read from dsh's own endmarkers, anchored at the end as dsh reads them. It is unknown for a command that reached its timeout (in a
session dsh moves it to a background job, or stops it when it cannot; the timeout printed is the one dsh applied:
the model's,
1000.5too, else 60 s, at most 600 s), was stopped, was killed by a signal or runs in thebackground, and for a result dsh's
--jsoncut at 8 KiB (the marker is in the cut part; the step says it wascut). Every unfinished end (
error,aborted,max-tokens,blocked, ...) is an error step.hermit/hosts/dsh.py(hermit deploy --host dsh) writesAGENTS.md,.dsh/skills, and a patch layer,.dsh/hermit.patch.yml, because dsh has no project file for MCP servers or skill folders. The layer holds theadapter's tool server (an
insertrow, 120 s per call, which fails the session's start when dsh cannot startit, as Codex's
requireddoes) and names.dsh/skills. A context past what dsh gives the model of a project'sinstructions (65,536 bytes, its 304-byte frame around
AGENTS.mdincluded) is refused. The acting agent passesthe layer with
--patch.AgentEvolver, the loader and the config acceptdsh.methods/dsh/is the new Evolver. It turns dsh's ownsandbox off inside the evolver sandbox (
permission_mode: danger-full-access), as Codex's is, and copies nologin (
home: []).hook_hosts.pygains adshwire).deepseek-official), and the user's~/.dsh/.env(or$DSH_HOME/.env) is copied into the session's home, as Codex'sauth.jsonis. This path was notrun (see Validation).
UNAVAILABLEgains dsh's codes for a provider that could not serve (RATE_LIMIT,SERVER,TIMEOUT,TRANSPORT,EMPTY_RESPONSE,STREAM_CLOSED), so the acting agent runs the task again. They match only at thestart of a
dsh: <CODE>:line.CliAgent.run):outcome()may now saystarted. When it does not,startedis as before (the CLIwrote a JSON line).
records.parse_logreads a dsh stream with dsh's reader. It read every stream other than Codex's as ClaudeCode's (a Pi stream too: fixed in Read a Pi acting agent's event stream as Pi's in parse_log and the AHE Evolver's texts #91).
Usagegainsunreported_calls, the calls whose tokens a CLI did not report, andrecords.price_usageleavessuch a usage's cost unknown (the budget counts it as such). Only dsh's reader sets it: Codex, Claude Code and Pi
are priced as on
main.agent-dsh.jsonl) instead ofagent-codex.jsonl(Pi's is missing on
maintoo: fixed in Read a Pi acting agent's event stream as Pi's in parse_log and the AHE Evolver's texts #91).mwmandRRSIEvolvers refusedshat setup: mwm would run it on dsh's own provider instead of its boundmodel (its session settings carry no
provider), and RRSI's read-only roles would run with all of dsh's tools(their settings are Codex's or Claude Code's).
pihas the same problem onmain: mwm and RRSI: refuse a learner backend other than Codex or Claude Code at setup #92 refusesevery backend but
codexandclaude, which coversdshtoo.login.evolver_problems, whichtrainruns before its first step, also checks a dsh Evolver's settings(
cli.dsh_problems), andhermit checknow reports it for an Evolver without a login too (it did only with one).hermit/isolate.py: only its config check (validate). dsh refuses alogin, as Pi does, and the acting agent'sdsh settings are checked there, as Claude Code's effort is on a login. No sandbox mechanics change (
.dshis aone-level folder). Pending: the core team's agreement, which the sandbox checklist asks for any
isolate.pychange.
agent-layers.md,model-backends.md,config-fields.md,task.md,evolver.md,methods/agent.md,custom-evolver.md,compatibility.md,installation.md, CHANGELOG), pluscommands.md,adapter.mdandmethods/ahe_adapter.mdforhermit deploy --host dshand the AHE hooks.hermit convert --target dsh(an adapter as a dsh plugin package).Validation
Run in an Ubuntu 24.04 container (arm64), as root with
--privilegedbecause the VM's AppArmor setting refusesunprivileged user namespaces. Python 3.12, bubblewrap, Node 24.22.2, and dsh 0.2.2-rc.1 from npm (its pi-ai
dependency resolved to 1.5.0). The toy runs ran on
207ea88. After it,1bbd57cadds one test (and a docstring'sline breaks);
f4792e0a recorded stream (promoted.jsonl) and, in an existing test, its case and two[stopped: …]results written by hand as dsh's bash tool renders them;d25958awidens the two timeout markers'patterns (round 8) and records
promoted.jsonlandbash.jsonlagain with the calls its tests read;a43698dcorrects wording only (round 9: comments, a docstring, the fixture README). The full suite ran again on the head,
a43698d.python -m pytest --integration,main(d5f0a64)rrsigroup installed,maina43698d)tests/core/test_dsh_agent.py(41 tests) andtest_deploy.py::test_dsh*hermit train+hermit evalon the docs' toy task, sandbox enforced, dsh acting and learning, scripted modelhermit checkok; train 0.0 → 1.0; eval: final adapter 1.0, none 0.0; no call of unknown costapi.fireworks.ai, and its probe found the network and the run's data closed; the key in no file the run wrotetests/core/test_isolate.py::test_a_rollout_sees_what_is_declared_and_nothing_elseandtests/methods/ahe/test_ahe_hook_runner.py::test_logs_that_cannot_be_written_lose_the_line_not_the_call.tests/fixtures/dsh/) are dsh's output unchanged, recorded as their README says.dsh-v0.2.2-rc.1and with the installed dsh.starts, every command fails with
SANDBOX_UNAVAILABLE, visibly.~/.dsh/.env.Known dsh behaviours (not changed here)
web_search: trueon DeepSeek's API, each search is a request of its own on the same key;as a call of unknown tokens);
usage at its end, after its tool calls ran, and a stopped session writes nothing more. Its cost is then that of
its earlier steps; a session stopped in its first step reports no usage, so the acting agent's record has an
unknown cost.
unavailable) keeps only its last attempt's usage, for every CLI: theacting agent's retry loop on
mainreports the last result's usage. dsh's outage codes reach that loop, and a dshsession can fail with one after billed steps. A separate change (it is shared code).
holds both kinds: such a run cannot be resumed on this head.
main) fails each step: dsh's own provider is notreached from the evolver sandbox.
.envfiles: dsh also reads a.envin the working directory. One that sets a reserved name (PATH,HOME,a proxy,
DSH_*) stops dsh before it writes any event; the acting agent reports that the session did not start.Codex's
.codex/config.toml.caps the output tokens instead, and compaction starts late on a small-context model.
AGENTS.mdwhen its file tools touch the folder. The acting agent'sadapter copy (
hermit/<id>/) holds one, identical to the context in the shipped templates.SANDBOX_UNAVAILABLE) andthe session still answers. With a task's data bound, SIGTERM ends the binding's bubblewrap, and dsh with it.
Known issues from review rounds 7 and 9 (low severity, not changed)
own message before any request): Claude Code's
claude_permission_modein a dsh agent's params becomes dsh'spermission mode; a
providerroute given with no model.DSH_HOMEwith a leading~(asexport DSH_HOME="~/x"leaves it): dsh expands it, hermit does not, so theuser's
.envis not copied for dsh's own provider, whose sessions then lack the key.sessionsfolder replaced by a link (by the session itself) is still resumed through it, fromwhere the session could read anyway.
agent.base: dshselecting dsh by its name; the dispatch ofhermit deploy --host dsh;agent: {name: cli, base: dsh}binding its model as dsh's; the dialecthermit checkand
trainrequire of a dsh model's backend (needs()); a one-key override of a dsh section, andagent.name: dshwith another
agent.baserefused; the patch layer written past a link a session left in a kept home; a toolserver name dsh refuses; a
--patch=<path>inargs; the clamp of a Responses total below the counts; a cutmessage, thought or tool result marked as cut; a tool other than bash, write, edit or an MCP tool kept as a
tool-call step; the outage codes matched only at a line's start; the key half of the web-search rule;
effort: offturning thinking off on dsh's own provider; (round 9) the exponent form of a timeout in both timeoutmarkers (
1e-7ms: JavaScript prints a timeout under 0.000001 ms so), which round 8's patterns match and whichread as exit 0 before them; only the decimal form (
1000.5ms) is tested.which is checked at load only on a login; one says only dsh's own provider can turn thinking off (dsh's pi-ai
routes can too; hermit declares none for a bound model).
Review
Documentation: pending.
agent.base: dsh(agent.name: dsh),evolver.base: dsh, thedshEvolver, adsh:settings section (permission_mode,image_input, ...),hermit deploy --host dsh, and theprovidersetting for dsh.hermit checkpasses on a dsh config.name, never its value. A grep found the Fireworks key in no record of the real run. The fixtures hold no key.
unavailable, so the task isrun again), a run stopped by SIGTERM (dsh exits 0), a run cut at the token limit, tools that did not start, a run
that started no session, a command moved to a background job at its timeout or killed at it (also at a timeout of
1000.5 ms), stopped, run in the background or cut (exit code unknown), and refused roles, login and container.
make("dsh").run, the acting agent'srun(with the deployed adapter's own stdio toolserver),
hermit trainandhermit eval. The stand-ins are only at the edges (the CLI, the model).tests/core/test_dsh_agent.py(andtest_deploy.py, the AHE, mwmand RRSI groups for their parts), named for the behaviour each holds;
tests/fixtures/dsh/is listed intests/selection.pyandtests/README.md, and its README says how each stream was recorded (dsh's outputunchanged). Each review round's fixes were undone one at a time to see their test fail (round 8's: the decimal
form of a timeout, not its exponent form; see Known issues). The stand-ins are a fake
dshthat replays a recorded stream and a scripted model on the loopback; tests that run the installed dsh aremarked
integrationand skip without it or without bubblewrap. Tests write totmp_path, and the check underbubblewrap to a directory of its own under
~/.cache/hermit-tests, removed after it.hermit.sandbox: the acting agent in the task sandbox(
isolate.py's worker), the dsh Evolver in the Trainer's evolver sandbox (a record from the Fireworks run:sandbox/learner-1.json,enforced: true, network isolated toapi.fireworks.ai, its probe passed). dsh's ownsandbox runs nested inside the run's for the acting agent (
workspace-write), and is off inside the evolversandbox, as Codex's is. The sandbox itself is unchanged;
isolate.pychanges only in its config check (see Corechanges), which the checklist counts as a core change: pending the core team's agreement.
was given the general, test and sandbox checklists):
03fafff: two reviewers, 23 findings. Fixed in7a076fb, each with a test that fails withoutits fix: bash results cut at 8 KiB read as exit 0; markers anchored with Python's
$; background commands readas exit 0; tools that did not start went unnoticed;
startedtrue for a run that wrote only an error;--add-dirbroke every command for a path through a link;agent.dsh.permission_modewas dropped; a gitrepository's skills and
AGENTS.mdreached an unsandboxed session; a workspace below a repository's root lostits deployed skills; no check of the 64 KiB instruction cap; usage of a step that answered without usage was not
counted; an
effortwithout a model was dropped;user_home()ignoredDSH_HOME; AHE had no dsh wire; mwm andRRSI accepted backends they cannot run;
parse_logread Pi's and dsh's streams as Claude Code's; tests fordsh's sandbox inside the run's, the context-tool portability loop and the deployed server's sandbox variables;
docstrings. Explained, not changed: what dsh does not report (compaction), the agent-writable patch layer, resume
in the same directory, and the
.envstop (now reported as no session). Docs: pending.7a076fb: two reviewers, 20 findings and questions (three found by both). Fixed in9d87553, each with a testthat fails without its fix: task data was linked, which dsh's search does not follow; web search with another
backend's model went to DeepSeek on
DEEPSEEK_API_KEY; calls without reported tokens were priced at $0.00; thedeployment marker could make a folder above the workspace the instructions' root; the deployed tool server did
not fail the session when it could not start; the size check left out dsh's 304-byte frame; the AHE texts named
Codex's stream for Pi and dsh; an effort level or permission mode dsh lacks failed every session instead of the
config; docstrings (the deploy command named no profile); tests for an AgentEvolver step, the deploy's limit and
a server name dsh refuses, and a recorded SIGTERM stream (
stopped.jsonl). Explained, not changed: dsh injectsa nested
AGENTS.mdwhen its file tools touch a folder (the templates' copies equal their context); thedeployed skills row replaces a
skill-filesystemconfig of the user's own (documented); the context window.The round-1 message overstated one change: a dsh usage error was already an error record per task, not a wrong
answer; it now stops the run.
9d87553: two reviewers, 16 findings and questions (two found by both). Fixed in2021944,each with a test that fails without its fix. One was a regression of round 2's own fix: its shared pricing rule
made Claude Code's results of refused requests (one turn, no tokens) cost unknown, and the AHE and RRSI merges
then dropped a step's known spend from the budget; it is replaced by the dsh-only
unreported_calls. The others:dsh's zstd transcripts escaped key scrubbing; the dsh learner was given an adapter's
AGENTS.mdas instructions;the dsh Evolver's settings were not checked at load, and YAML's bare
offwas notoff; the search rule went bythe backend's name; a missing
context_windowfailed every request on vLLM without saying why; the task-databinding had no test CI runs; wording (the deployed server row, a run dsh never started, the size check's scope,
the linked-data warning). Explained, not changed: dsh's refusal of every command without bubblewrap or Landlock
(not reproduced: a container without privileges fell back to Landlock); a session with bound data stopped by
SIGTERM ends without an event.
2021944: two reviewers, 14 findings, questions and checklist items (two found by both). Fixed inf4b7b09,each with a test that fails without its fix: a step dsh retried after a refused request lost its cache tokens
and was priced as known (dsh drops the step's cache counts; recorded:
retried.jsonl), which also corrects round3's account of retries; the dsh Evolver's checks missed the method named with its params and refused an effort
with the run's own model (they now run on the Trainer's params, in
hermit checkand beforetrain's firststep); efforts dsh's own provider refuses passed;
evolver.dsh.instruction_fileswas dropped, not refused; thecaller's
DEEPSEEK_SEARCH_BASE_URLmoved dsh's search without the sandbox; the context-window hint cut dsh'serror. Explained, not changed: a home of compressed transcripts from earlier commits; an Evolver with no model
reference (as Pi's on
main);agent.dsh.instruction_files, like other free-form settings, does not reach theacting agent; a successful result keeps an unread
failuretext, as Pi's does.f4b7b09: two reviewers, 14 findings, questions and test gaps (one found by both). Fixed inf1a92ea, each with a test that fails without its fix: round 4's unknown cost for a retried step made a wholesession's spend invisible to a USD budget, while its tokens are known and hermit prices a cache write as input
(now priced); the caller's
DEEPSEEK_BASE_URLmoved dsh's own provider without the sandbox; the Evolver's checkrefused its own values, and the acting agent's
instruction_files: false, which would cut the adapter's context,was dropped; tests now pin the session's tool row (
failOnStartupError, its 30-minute call timeout), every rowDSH_OFFturns off and all six outage codes; wording:effort: offasks a bound model for no level (itsserver's default), and the context-window hint keeps dsh's error line as far as it fits. Explained, not changed:
a Responses server's own
total_tokensis taken as pi-ai gives it; a session with an unreported call has anunknown cost, which a USD budget does not count (as any unknown cost); a deployed tool server's stderr reaches
dsh's (not reproduced as a problem).
f1a92ea: two reviewers, 11 findings (none found by both); the fixes change claims and tests,no behaviour. Fixed in
37aee8c: tests, each failing without what it checks: a session's ownenvreaching dshhad no test; the integration test of dsh's own sandbox now shows a command's write outside the workspace refused
(it passed with the sandbox off); a bash call that failed to run has no exit code, and a session whose folder, or
the folder it lies in, is a link is not resumed (neither was tested); and wording, in the reader's account: a
stopped session's last step is not counted (its docstring said it was a call of unknown tokens), dsh's own
adapter, not pi-ai,
drops a cache count of 0, and a retried step's price is never less only for a price whose cached input is no
dearer than its input, and on the Responses API only as far as the server's
total_tokensholds the tokens; astale test docstring. Explained, not changed: a sandbox's grant of
DEEPSEEK_BASE_URLdoes not reach dsh (seeLayer 2); a task run again after an outage keeps only the last attempt's usage, for every CLI (see Known dsh
behaviours).
37aee8c: a workflow of five reviewers, one lens each (usage and cost; settings and entrypoints; the session at run time; parity with Pi and shared code; tests and claims), each finding then reproduced
by a separate agent told to refute it. 34 findings (27 distinct), all confirmed, none a wrong result in a correct
configuration. One was rated medium: no test guarded which steps of dsh's stream count as calls, the rules a USD
budget rests on; tested in
1bbd57c. One more, rated medium by its reviewer and low by the agent thatreproduced it (the code was right), was changed at the author's request: no test held a command dsh moves to a
background job at its timeout, what a timeout gives in a session unless dsh's job limit is reached, and only one
check keeps it from reading as exit 0.
f4792e0records one (promoted.jsonl) and tests it, and also acommand stopped from outside the call (
[stopped: <reason>]), whose two results are written by hand as dsh'sbash tool renders them: no headless session writes one. The Pi fixes this branch carried moved to Read a Pi acting agent's event stream as Pi's in parse_log and the AHE Evolver's texts #91 and mwm and RRSI: refuse a learner backend other than Codex or Claude Code at setup #92
(
207ea88), so that this PR changes only dsh, and the check under bubblewrap keeps its home under~/.cache/hermit-tests. Corrected in this description: its claims that mwm and RRSI both ranpiwithout itsmodel (RRSI ran it on its model, with role settings Pi does not read), that round 6's fixes were all tests, that
a stopped session's cost is always known, that the acting agent's check at load is as Claude Code's effort check
(that one runs at load only on a login), and its list of usage dsh does not report (web searches were missing).
The other low ones are not changed (the author's call): see Known issues from review rounds 7 and 9.
f4792e0: two reviewers, one on the test and its recorded stream, one on the claims, eachfinding then reproduced by a separate agent told to refute it. 6 findings (5 distinct), all confirmed, one in
part. Two were medium, both fixed in
d25958awith recorded results: dsh prints the timeout it applied to acommand, the model's when it gave one (
1000.5), and both timeout markers were matched only at a whole numberof milliseconds, so a command still running at such a timeout read as exit 0, and one killed at it that trapped
the signal as its trapped exit; and the marker alone, what a command that printed nothing yet gets, was
untested. The low ones are corrected: two in this round's own text (a docstring's order of dsh's markers, and
that the
[stopped: …]results are written by hand), and this description's diff size, stale since1bbd57c(
f4792e0added a file).d25958a: two reviewers, one on the code and its tests, one on the claims, each finding thenreproduced by a separate agent told to refute it. 6 findings (4 distinct), all low, all confirmed. Three were
wording this change's own text got wrong, corrected in
a43698dand here: dsh prints the timeout it applied(60 s when the model gives none, at most 600 s), not always the model's;
DSH_NO_OUTPUT's comment against abackground marker that stands alone; round 8's account of its low findings. The fourth, the exponent form of a
timeout untested, is not changed: see Known issues.
207ea88and the run records it cites; the parts1bbd57c,f4792e0,d25958aanda43698dchange, against those commits.🤖 Generated with Claude Code