diff --git a/plans/USAGE-PATH-TESTS.md b/plans/USAGE-PATH-TESTS.md new file mode 100644 index 00000000..2bbbcb11 --- /dev/null +++ b/plans/USAGE-PATH-TESTS.md @@ -0,0 +1,191 @@ +# Usage-path tests: an explorer writes the test once, CI runs it forever + +Status: **design draft, 2026-09-03.** Nothing here is built. The draft records +the minimum shape of a test that exercises a human critical path against real +dependencies, authored once by an agent and run afterwards with no agent +present. It is deliberately not a workflow engine. + +## 0. The gap this closes + +The three gates, the Go suites and the box-image smoke test prove that the +code is coherent and that a box boots. None of them sees a real control plane, +a real VM, a real vendor login or a real Lody turn. The live scripts under +`tools/e2e/` do, but each one was written by hand for one incident, takes a +person to run, and is silently stale the next time the surface it drives +changes (`tools/e2e/credentials.mjs` still calls a `blitz-cred token` verb the +binary no longer has). + +The broker retirement (plans/BROKER-RETIREMENT.md) was verified by an agent +that brought environments up, poked them, wrote checks and tore them down. +That work was thrown away when the session ended. The point of this plan is +that the second time a path needs verifying, nobody has to do that again. + +## 1. Locked shape + +One artifact, one loop, one authoring rule. + +1. **A job is a directory in the repo.** A manifest and an entry script. The + script drives the product the way a person does and asserts at the surface + a person sees. It is an ordinary test after it is admitted. +2. **The runtime is a sandbox with a lease and an allowlist.** Allocate, + mint, fence, run, collect, revoke, destroy. Nothing survives a job except + its artifacts. +3. **An explorer agent writes each job once**, in the same sandbox, lease and + allowlist the job will run in, and the job is admitted only after it passes + once more with no agent present. + +There is no DAG, no durable execution, no step debugger, no trace recorder +and no undo. Each of those was considered and cut; ยง6 says why. + +## 2. The job + +``` +jobs/// + job.toml + run # the entry script; any language the image carries + artifacts/ # written by the run; collected by the runtime +``` + +`job.toml` declares five things and nothing else: + +```toml +# The human path, in one sentence. The explorer worked from it; a reader +# checks the assertions against it. +path = "A member signs in, creates a workspace, and reaches a terminal tab." + +[sandbox] +image = "blitz-box" # what the run executes inside +size = "small" # driver-specific + +[secrets] +# Names only. The runtime mints a lease per name for this job id. +names = ["GH_PAT", "ANTHROPIC_OAUTH"] + +[network] +# Egress the path genuinely needs. Everything else is refused. +allow = ["cp.example", "api.anthropic.com", "github.com"] + +[budget] +wall = "20m" # also the lease TTL +retries = 1 +money = "0.10 EUR" # for jobs that create cloud resources +``` + +The entry script exits 0 or non-zero and may print lines the runtime +classifies: + +- `TRANSIENT: ` โ€” retry within the budget. +- `ENVIRONMENTAL: ` โ€” a missing tool, token or quota. Escalates to a + person; it is never a product finding. +- anything else non-zero โ€” a real failure of the path. + +The script asserts what the person would see: HTTP status and body, terminal +output, files on disk, browser state through a headless browser when the path +has a UI. Assertions against internal state are allowed only as a second line +of evidence, never as the verdict. + +## 3. The loop + +``` +allocate a fresh sandbox from the image, labelled with the job id and a TTL +mint one short-lived credential per named secret, scoped to the job id, + expiring at the budget; the sandbox receives the lease, never the root +fence apply the allowlist; direct egress is refused +run check out the SHA under test inside the sandbox; execute `run` +collect copy artifacts out +revoke the lease, whether the run passed or not +destroy the sandbox +``` + +An independent reaper destroys any sandbox and revokes any lease whose TTL +passed, so a dead runner cannot leave a bill or a live credential. That is +the whole runtime. It runs as one CI step per job, jobs in parallel. Debugging +is rerunning the job, or opening a shell in a sandbox with the same lease and +allowlist. + +The allowlist is the leak guard as much as the dependency list: a leased +token can only travel to hosts the job declared, so an explorer with a real +credential cannot post it anywhere the path does not go. The simplest portable +fence is an egress proxy the sandbox must use, with direct egress blocked at +the network namespace. + +## 4. The explorer + +Once per feature, an agent runs in exactly the environment the job will run +in, with the human path as its brief and the acceptance list as its goal. Its +only deliverable is the job directory. It may use real credentials because +the lease bounds how long a leak is useful, the allowlist bounds where it can +go, and the sandbox bounds what it can break. Remove any one of the three and +either real secrets are off the table or an agent cannot be allowed to write +the test. That is why these three are the floor and nothing smaller works. + +Admission has one rule: the job passes once more with no agent present before +it merges. + +When an admitted job fails later, the runtime classifies by the lines above. +Only a real failure reaches an agent, with the artifacts, the diff since the +last green run and the brief. The agent must answer with one of four verbs: +patch the job and rerun, skip with a written reason, file a product bug, or +abort. Patches are commits, so drift has a history a reviewer can read. + +## 5. What this repository already has + +- **Sandbox.** The box image is one, Docker included, and the microVM path + is a faster one. The machine API (`POST /machines/:id/provision`, + `/stop`, `DELETE`) already allocates and destroys real machines with + labels, and refuses to destroy what a person made. +- **Lease.** The org-credential plane mints scoped, expiring tokens with an + audit row and a grant check on every ask. `blitz-cred api-token` is the + local primitive. +- **Execution environment.** The image carries node, python3, git, gh, + docker and both vendor CLIs. + +Missing: + +- a per-job egress allowlist inside the box (an egress proxy plus a default + deny in the box's network namespace); +- the `jobs/` directory and the manifest above; +- the explorer brief, and the admission replay in CI. + +## 6. Deliberately absent + +- **No DAG or durable execution.** A job is a script; composition is a + script calling scripts. A graph runner would be the largest piece of code + in the system and would test nothing. +- **No trace recorder.** The explorer writes the job on purpose. The + admission replay is what catches what it forgot. +- **No undo.** Teardown is destroy, revoke, reap. Idempotent steps and + compensation are what a runner needs when environments are precious; these + are not. +- **No stepping.** Rerun the job. +- **No evidence diffing.** Artifacts on disk. Add comparison when two runs + ever need comparing. + +## 7. The first jobs + +Each of these is a path a person walks today and that no gate covers: + +1. sign in, create a workspace, reach a terminal tab through the tunnel; +2. run one Lody turn with Claude signed in natively, and one with Codex; +3. push to GitHub through the box's git credential helper; +4. enroll a box, rotate its bearer, and call `/agent/*` with the new one; +5. request a box update from the box plane and watch the new image land; +6. boot the previous image on a state volume, then this image on the same + volume, and assert the home was repaired (the broker-era incident). + +Jobs 1 through 5 need a control plane that is not canary. A staging Worker +cloned from canary's `wrangler.toml` under a different Worker, D1 and R2 name +is enough; the canary workflow only forbids a branch reaching the shared +canary Worker. Job 6 needs Docker and nothing else. + +## 8. Open decisions + +- Where the staging Worker lives (same Cloudflare account as canary, or its + own) and who holds its secrets. +- Whether the egress fence is a proxy in the box or nftables on the host. A + proxy is portable to every substrate; nftables is invisible to the job. +- The reaper's home: the control plane already sweeps orphaned machines, so + the job-scoped TTL may be a label it honours rather than a new service. +- Whether explorer runs are allowed against canary at all, or only against + staging. The lease and allowlist make it safe; the question is blast radius + on shared data.