Skip to content

codex-plan.md

Latest commit

 

History

History
184 lines (139 loc) · 9.95 KB

codex-plan.md

File metadata and controls

184 lines (139 loc) · 9.95 KB
Raw

Run harness evolution with your coding plan

Use your Codex / ChatGPT plan to run both the acting agent and the learner in Hermit. You can try methods such as GEPA, SkillOpt, ACE and RRSI Adapter without configuring a separate model API key or API billing account. Hermit runs the method's evaluation and selection loop; Codex supplies the model sessions that act and learn.

This tutorial uses the existing RRSI Adapter smoke run on OfficeQA: four training questions, two trials per question, one round of two candidate adapters, then four held-out questions. It checks the integration rather than establishing a performance gain. It still needs the benchmark data and execution environment described below.

Start with a coding-agent prompt

Open the Hermit checkout in your coding agent and give it this prompt. Replace the model placeholder with a model available to your Codex login.

Help me run a small harness-evolution experiment using my Codex / ChatGPT plan.
Use <my Codex model name> for both acting and learning, without a model API key.

Read AGENTS.md, docs/tutorials/codex-plan.md and the RRSI Adapter example.
Check the prerequisites, then use examples/officeqa/rrsi_adapter/smoke_luna.yaml.
Keep its one-round protocol. Set acting concurrency to 2 and the training time
budget to 2 hours. Use the same model settings for check, train and evaluation.

Use the existing data-preparation scripts, container and egress-proxy setup.
If the OfficeQA dataset or another prerequisite is missing, tell me exactly what
I need to provide. Do not substitute synthetic data for the benchmark.

Run the checks, train, and evaluate both the seed and final adapter. Inspect
whether the round completed, which candidate was selected, and any errors.
Report the commands, output paths and results. Keep generated artifacts local;
do not commit or push anything.

The following sections explain the setup and commands behind that prompt.

1. Prepare Codex and the benchmark

Use a Linux machine with Python 3.12+, Git, bubblewrap and Docker for this example. RRSI's proposer and search roles use Hermit's Linux sandbox; acting sessions run in Docker. Complete Installation and install the two dependency groups from the repository root:

python -m pip install --upgrade "pip>=25.1"
python -m pip install -e . --group rrsi --group skillopt

rrsi supplies the search method; skillopt supplies OfficeQA's shared data parsing and task utilities.

Install Codex CLI, then sign in with ChatGPT on the machine that will run Hermit. Hermit currently needs Codex's file-based login cache:

codex -c 'cli_auth_credentials_store="file"' login
codex login status

Confirm that the authentication method is ChatGPT. Hermit reads auth.json under CODEX_HOME, or ~/.codex when it is unset; a login held only in the OS credential store is not enough. Hermit manages per-session copies and refreshes through its shared login directory. See OpenAI's authentication guide for sign-in and credential-storage options.

Complete these existing OfficeQA setup steps once:

  • Prepare the data: obtain the gated dataset, then run prepare.py with the CSV and parsed corpus paths. This writes the task splits and initial skill used below. Dataset access is separate from model authentication.
  • Run in the Codex harness and A model on a ChatGPT login: build the Linux Codex image, create the internal Docker network, and start the login egress proxy. The example names officeqa-codex:0.164.0, oq-sandbox and oq-egress-login. The image needs a Linux Codex distribution matching Docker's architecture, even if you normally use Codex on another operating system.

Then prepare RRSI's seed and trial rows:

python examples/officeqa/rrsi_adapter/make_seed.py
python examples/officeqa/rrsi_adapter/make_rows.py

2. Check, train and evaluate

Run from the repository root. Choose a model your login can actually use; the checked-in recipe's gpt-5.6-luna is the model used for its reported experiment, not a requirement to use your plan.

HERMIT_CODEX_MODEL='your-codex-model-name'

hermit check --config examples/officeqa/rrsi_adapter/smoke_luna.yaml --no-probe \
  agent.model.name="$HERMIT_CODEX_MODEL" evolver.model.name="$HERMIT_CODEX_MODEL" \
  agent.concurrency=2 trainer.budget.hours=2

hermit train --config examples/officeqa/rrsi_adapter/smoke_luna.yaml \
  agent.model.name="$HERMIT_CODEX_MODEL" evolver.model.name="$HERMIT_CODEX_MODEL" \
  agent.concurrency=2 trainer.budget.hours=2

hermit eval --config examples/officeqa/rrsi_adapter/smoke_luna.yaml --adapter seed_adapter \
  agent.model.name="$HERMIT_CODEX_MODEL" evolver.model.name="$HERMIT_CODEX_MODEL" \
  agent.concurrency=2

hermit eval --config examples/officeqa/rrsi_adapter/smoke_luna.yaml --adapter final \
  agent.model.name="$HERMIT_CODEX_MODEL" evolver.model.name="$HERMIT_CODEX_MODEL" \
  agent.concurrency=2

check --no-probe checks prerequisites without a task rollout; it does not prove model access or learner execution. Training normally evaluates H_0 and two candidates: 24 acting rollouts, plus candidate liveness smokes and the proposer, analyst, digester and critic sessions. Each evaluation command adds four acting rollouts. A small task split can still consume substantial plan usage. The two-hour training budget can end the run before its round finishes; probes and separate evaluations are outside that training budget.

Outputs are under examples/officeqa/rrsi_adapter/runs/smoke-luna-rrsi/. Inspect:

  • checkout/ADAPTER.md and any additional learned files;
  • step_NNNN/output/report.md for candidate screening, round decisions and the selected incumbent;
  • workspace/state/rrsi/frontier.json for the incumbent and round trajectory;
  • eval/seed_adapter/summary.json and eval/final/summary.json for held-out scores and execution errors.

Check that the round completed before treating the output as a completed RRSI smoke. final is the incumbent H_T; an interrupted run can return its last incumbent. A run that completes with no accepted candidate is also valid. Keep model and task settings fixed when comparing seed and final scores.

For a separate experiment, change both trainer.run_name and trainer.tree_dir; apply both overrides consistently to its training and evaluation commands. An unfinished run resumes automatically. Keep generated records and adapter snapshots local, as described in Contributing.

How this works for other methods

There are two learner configurations, depending on how the method calls its model:

Method's learner Configuration Examples
Requests text replies for reflection or revision llm.cli: codex with login: chatgpt; each request becomes a Codex session in a sandbox of its own GEPA, SkillOpt, ACE, Continual Harness
Edits files as an agent evolver.base: codex and evolver.model.login: chatgpt; sessions run in the method's workspace Harness², Meta-Harness Adapter, AHE Adapter, RRSI Adapter

For the first form, the model settings look like this; use the method's existing recipe for its remaining fields:

llm:
  cli: codex
  model: your-codex-model-name
  login: chatgpt
  effort: medium
  web_search: false
  isolation: sandbox     # the default: a bubblewrap sandbox of its own, the login's hosts through a proxy
  timeout: 3600

isolation: container with container: {image, network, proxy} runs each session in that image instead.

For the second form, keep the method's evolver.name and parameters, and configure its Base Agent and model:

evolver:
  name: rrsi_adapter
  # Keep the recipe's method-specific params here.
  base: codex
  model: {name: your-codex-model-name, login: chatgpt, effort: medium}

The acting agent also needs agent.base: codex and agent.model.login: chatgpt, as in the smoke configuration. Changing only the learner does not change an API-based actor. Use a complete recipe for the task's preparation, scoring, container and sandbox settings. For example, ACE already provides examples/officeqa/ace/smoke_codex_luna.yaml; GEPA and SkillOpt have run_codex_luna.yaml recipes.

Plan-backed sessions use the account's available models and usage limits. Hermit reports unknown dollar cost when there is no per-token price; use time/token budgets and inspect the shared plan status described in the login setup. Extra services still need their own setup: ReHarness's embedding model, for example, is not provided by this Codex text-model route.

What about a Claude Code plan?

A Claude subscription (Pro, Max, Ultra, Team or Enterprise) runs the same way through Claude Code, with no API key: claude setup-token once, the token exported as CLAUDE_CODE_OAUTH_TOKEN, and login: claude in place of login: chatgpt with Claude Code as the CLI (agent.base: claude with its agent.claude section, llm: {cli: claude, ...}, evolver.base: claude) and, for a container, the Claude Code image and egress proxy; start from the Claude configs named here, not from this page's Codex smoke with its login changed. Hermit copies no credential: the token passes by name with each session. Acting and learning on a Claude subscription has the sign-in, the plan (~/.cache/hermit/claude-login/plan/), the boundaries and which methods' roles go which way; examples/claude_subscription/ is a small run on nine tasks, and OfficeQA's skillopt/run_claude_haiku.yaml is run_codex_luna.yaml with the agent and the models changed.

Continue with Choose a method, the OfficeQA recipe, or Bring your task to use your own benchmark.