Skip to content

RFC: Harness evolution #261

Open
2 of 7 issues completed
Open

Description

@GoDiao

Status

Long-term. This RFC tracks how DreamCoder's agent runtime (the harness: tools, prompts, request handling, agent loop) evolves. It stays open; concrete changes get their own issues and PRs and are linked here.

Summary

DreamCoder's CLI is derived from Claude Code 2.1.92, which was tuned for Claude models. Most DreamCoder users run third-party models through OpenAI-compatible endpoints. This RFC is about making the harness work well for them: fewer tokens and turns per task, and behavior that fits the model actually in use.

Why

FrontierHarness Eval ran 62 terminal tasks across 22 harnesses and 5 open-weight models served through an OpenAI-compatible provider.

  • Pass rates sat between 51% and 62%, and most differences were within the confidence intervals.
  • Cost and run time differed a lot. Claude Code averaged about 4.2M input tokens and 41 minutes per task; Codex CLI about 3.0M and 27 minutes.
  • Codex CLI, Pi and ZCode were on the pass-rate/cost Pareto front.

A first source comparison of Codex CLI, ZCode and DreamCoder is in docs/features/harness. Its findings are leads to check, not settled conclusions.

How we work

  • Measure first. Build a fixed set of tasks and record tokens, turns, cache hits and pass rate per run. Every change reports numbers before and after on the same set.
  • One concern per change. Each change gets its own issue, links back here, and says what it is expected to improve.
  • Keep the record in the repo. Findings and decisions go into docs/features/harness; this issue keeps the list of open and finished work.

Areas

Candidate directions, none verified yet:

  • Request path through the local proxy: what reaches the provider, and what it can cache.
  • Size of the fixed part of every request: tool definitions and the system prompt.
  • Text written for Claude that other models also receive.
  • Tool execution: editing, reading, and recovery from errors.
  • Per-model knowledge: context window, output limit, reasoning settings.

Work items

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestrfcRequest for comments / design discussion

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions