From 7dcd640a979acbbe958fdaee86f38af0a4245ee4 Mon Sep 17 00:00:00 2001 From: MendixMau Date: Tue, 27 Oct 2026 08:51:42 +0100 Subject: [PATCH 1/2] process(roadmap): RM-02 measure build speed and token cost per release Run 2 of the ProcureFlow cook-off found a regression no release caught: 184 commits since 247729fd added per-exec checks (lint, score, approval refusal, row-as-unit) and removed none. Arm C: 0.93 scripts/row (cloud 0.34), lint+score 18.3 min unread, verify-module 0 runs, docker reload broken, stopped at row 121/147 after 10 h 50 min; 399M lead input tokens, 40k fixed load per call with 0 Skill/MCP calls. RM-02 parks a per-release benchmark, cost-and-reader per check, cadence over repetition, quiet output and lean sessions. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01XM8AzTxUsYcriPXQ58ZPEV --- process/roadmap.md | 66 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 66 insertions(+) diff --git a/process/roadmap.md b/process/roadmap.md index b6b5c8ea..1d6d7f94 100644 --- a/process/roadmap.md +++ b/process/roadmap.md @@ -121,3 +121,69 @@ something to generalise from a single session. vendor's docs are unreachable from the sandbox these assessments were written in. Cost is metered per spec run, and nobody has measured what one of this pipeline's stages costs. Both figures are for the field run to supply. + +--- + +## RM-02 — Every toolkit release is measured for build speed and token cost, so a regression is caught in a week, not a cook-off + +**Status:** PARKED — the evidence is in; the measuring harness does not exist yet. + +**The regression nobody spotted.** Between the copy a cloud build used (`247729fd`, 2026-09-21) +and `c9dedb11`, 184 commits landed, 36 of them in `project-bin/exec.sh` (819 → 2,289 lines). Each +added a check after a real failure — lint per exec (`8d70fbda`), score per exec (`a7221405`), the +chat-approval baseline refusal (`b6a13380`), the row as the unit of work (`a7653197`) — and none was +ever removed or re-costed. No release measured what the sum did to a build. A cook-off did +(run 2, ProcureFlow, 2026-10-26/27, arm C = mxcli + this toolkit at `c9dedb11`): + +| Measured | Value | +|---|---| +| Scripts per build-plan row | 0.93 (the cloud build: 0.34) — every row paid the full exec chain | +| Lint + score after every exec | 18.3 min; the ratchet read ROSE on 123 of 147 rows and nobody read it — while 6 real SEC010 warnings sat in it | +| Checks that never ran | `verify-module`, `gate-check`, the LOOK ledger: 0 runs (nothing calls them) | +| `mxcli docker reload` | failed every time (curl not in the JRE image) → a container restart per change | +| Outcome | stopped by the user after 10 h 50 min at row 121 of 147, no acceptance test; the HAM arm finished 137/137 and 12/12 in 602 min | +| Lead tokens | 2,878 model calls, median 139,895 tokens of context per call, 399 M input tokens in total | +| Fixed load per call | 40,161 tokens, incl. 188 skills, 7 user plugins and 4 MCP servers — **0** Skill and **0** MCP calls in the whole run | + +**Assets (today):** + +| Asset | Path | State | +|---|---|---| +| Fixes under test (arm D of the cook-off) | `project-bin/reload.sh` (hot reload via `--direct`), `project-bin/module-close.sh` (lint, score, check, verify-module, LOOK once per module), `project-bin/row-groups.sh` + an `exec.sh` gate (low-risk rows grouped into one script, high-risk rows alone), `skills/iterative-build-loop.md` (row groups, diff only on overlap) | local branch `run3-speed` (9b2774db, 78e34c93), not pushed; measured in arm D before any merge | +| Lean session switches | `claude -p --setting-sources project,local --strict-mcp-config [--mcp-config .mcp.json]` → 37,536 tokens per call (from 40,161); with `--disable-slash-commands` 33,329 | measured once, one call | +| Run analysis | `procureflow-ham-runs/OBSERVER-FINDINGS.md` O-49 … O-56; `personal-toolkit/research/2026-10-27-run2-toolkit-speed/` | written | + +**What to build:** + +1. **A speed benchmark per release.** A small fixed reference build (a slice of a real build plan, + ~10 rows, one module) run headless on each release candidate. Record: execs per row, instrument + seconds per exec, minutes per row, model calls, median context tokens per call, fixed load per + call. Compare with the previous release; a rise of more than 15% on any of these blocks the + release until it is explained in the CHANGELOG. +2. **Every check carries its cost and its reader.** A new step in `exec.sh` (or any per-exec hook) + states in its header: seconds per exec, tokens of output per exec, and who reads the result and + when. A check whose output no step reads is moved to a cadence (per module, per checkpoint) or + removed. Review the list at each weekly toolkit review; removing is a normal outcome. +3. **Cadence over repetition.** Per exec: only what protects the model (check, apply, mxbuild gate). + Per module: lint, score, verify-module, LOOK (`module-close.sh`). Per checkpoint: docker + check/build, the test suite. A check that "must" run is called by a gate, never only by a prompt. +4. **Quiet by default.** `exec.sh` prints the verdict and the errors; the long form goes to a log + file. Every line printed is re-read on every later model call until the next compaction. +5. **Lean sessions by default.** Headless and build sessions start with the project's settings only: + the project's own MCP servers (`.mcp.json`) stay, so teams that use an MCP server keep it; user + plugins, user skills and connectors a build never calls are left out. + +**Start when:** arm D's numbers are in (same plan, `run3-speed`, solo) — or the next toolkit release, +whichever comes first. + +**Exit bar:** the benchmark runs on every release and its table is in the release notes; `exec.sh` +lists cost and reader for each step; lint/score/verify-module run at module close under a gate +(not a prompt), with a non-zero run count in a field build; a headless build's fixed load per call +is reported and below 35k tokens. + +**Explicitly out of scope:** + +- **Removing lint, the score or the mxbuild gate.** They find real defects (the 6 SEC010 above). + The fix is when they run and who reads them, not whether they exist. +- **Tuning for one arm of the cook-off.** The benchmark measures the toolkit on a fixed plan; it is + not a race against HAM or the cloud build. From 17f75bd256562a2c4dd362e8cecb4aadb6f5a521 Mon Sep 17 00:00:00 2001 From: MendixMau Date: Tue, 27 Oct 2026 08:52:06 +0100 Subject: [PATCH 2/2] process(roadmap): RM-02 keeps lint per exec, new warnings only Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01XM8AzTxUsYcriPXQ58ZPEV --- process/roadmap.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/process/roadmap.md b/process/roadmap.md index 1d6d7f94..cda2b705 100644 --- a/process/roadmap.md +++ b/process/roadmap.md @@ -164,8 +164,10 @@ ever removed or re-costed. No release measured what the sum did to a build. A co states in its header: seconds per exec, tokens of output per exec, and who reads the result and when. A check whose output no step reads is moved to a cadence (per module, per checkpoint) or removed. Review the list at each weekly toolkit review; removing is a normal outcome. -3. **Cadence over repetition.** Per exec: only what protects the model (check, apply, mxbuild gate). - Per module: lint, score, verify-module, LOOK (`module-close.sh`). Per checkpoint: docker +3. **Cadence over repetition.** Per exec: what protects the model (check, apply, mxbuild gate) and + lint — kept per exec by decision (2026-10-27: deferring it risks missing findings), but it prints + only the NEW warnings in the documents this script wrote, so it is read, not skimmed past as ROSE. + Per module: score, verify-module, LOOK (`module-close.sh`). Per checkpoint: docker check/build, the test suite. A check that "must" run is called by a gate, never only by a prompt. 4. **Quiet by default.** `exec.sh` prints the verdict and the errors; the long form goes to a log file. Every line printed is re-read on every later model call until the next compaction. @@ -177,7 +179,7 @@ ever removed or re-costed. No release measured what the sum did to a build. A co whichever comes first. **Exit bar:** the benchmark runs on every release and its table is in the release notes; `exec.sh` -lists cost and reader for each step; lint/score/verify-module run at module close under a gate +lists cost and reader for each step; new lint warnings are shown per exec and acted on; score/verify-module run at module close under a gate (not a prompt), with a non-zero run count in a field build; a headless build's fixed load per call is reported and below 35k tokens.