Repository navigation
Description
Status
Long-term. This RFC tracks how DreamCoder's agent runtime (the harness: tools, prompts, request handling, agent loop) evolves. It stays open; concrete changes get their own issues and PRs and are linked here.
Summary
DreamCoder's CLI is derived from Claude Code 2.1.92, which was tuned for Claude models. Most DreamCoder users run third-party models through OpenAI-compatible endpoints. This RFC is about making the harness work well for them: fewer tokens and turns per task, and behavior that fits the model actually in use.
Why
FrontierHarness Eval ran 62 terminal tasks across 22 harnesses and 5 open-weight models served through an OpenAI-compatible provider.
- Pass rates sat between 51% and 62%, and most differences were within the confidence intervals.
- Cost and run time differed a lot. Claude Code averaged about 4.2M input tokens and 41 minutes per task; Codex CLI about 3.0M and 27 minutes.
- Codex CLI, Pi and ZCode were on the pass-rate/cost Pareto front.
A first source comparison of Codex CLI, ZCode and DreamCoder is in docs/features/harness. Its findings are leads to check, not settled conclusions.
How we work
- Measure first. Build a fixed set of tasks and record tokens, turns, cache hits and pass rate per run. Every change reports numbers before and after on the same set.
- One concern per change. Each change gets its own issue, links back here, and says what it is expected to improve.
- Keep the record in the repo. Findings and decisions go into
docs/features/harness; this issue keeps the list of open and finished work.
Areas
Candidate directions, none verified yet:
- Request path through the local proxy: what reaches the provider, and what it can cache.
- Size of the fixed part of every request: tool definitions and the system prompt.
- Text written for Claude that other models also receive.
- Tool execution: editing, reading, and recovery from errors.
- Per-model knowledge: context window, output limit, reasoning settings.
Work items
- Benchmark task set and baseline numbers
- Proxy layer (review):
- Harness: proxy request path fixes #262 Request path fixes (fix(proxy): fix the basics of the OpenAI-format request path #266)
- Harness: provider form for extra headers, and before/after measurement for #262 #267 Provider form for extra headers and query parameters; before/after measurement for Harness: proxy request path fixes #262
- Harness: context overflow, errors and stream robustness in the proxy #263 Context overflow, errors and stream robustness (fix(harness): recover from provider errors, broken streams and context overflow #268)
- Harness: treat a direct Anthropic stream that ends without stop_reason as incomplete #269 Direct Anthropic stream that ends without stop_reason
- Harness: per-model capabilities in the proxy #264 Per-model capabilities
- Text layer (review):
- Harness: correct the text layer for DreamCoder and non-Claude models #270 Correct the text layer for DreamCoder and non-Claude models (sub-issues Harness: load AGENTS.md and DREAMCODER.md as project instructions #271–Harness: attribute commits and PRs to DreamCoder, with a setting to turn it off #280)
- Harness: keep session-start context in the history across CLI restarts #281 Keep session-start context in the history across CLI restarts
Activity