[MaxGhenis]

Stacked on #240. It merges into v2-microcosm-batches now that the cheap batch 1 and the free-plan run, which both ran from that branch at 5bbd571, have finished (2026-10-24 05:31Z and 05:42Z).

What

The calculator's features become independent switches, so a study can change one at a time. Max, 2026-10-23: "measuring this as an experiment could be publishable", "do all these tests with a cheap model", "and tool v3".

feature	what it does
names	a name with spaces, or a benchmark output key, is read as its engine variable (tool v2)
labels	a fact under any label the prompt shows it under (built from the prompt's own renderer: "gross wages and salaries", "usual weekly hours worked", the v2-pilot relabelings, the Marketplace plan line, names with spaces) is read as its variable. Stated hours are also set where the reference sets them (PE_INPUT_ALIASES), and facts the reference drops (PE_DROPPED_INPUTS) are ignored with a note
placement	an input set on another entity moves to the one unit holding its people (tool v2)
overrides	a realized contribution is read as _desired, and an input that replaces a formula is flagged (tool v2)
explain	explain traces up to three variables (tool v2)
concise	explanations are brief unless detail is "full"
flat	the model describes household (state, people with role and facts, household facts), and the tool builds the units Scenario.to_pe_household builds

Arms: pe2 is unchanged (names, placement, overrides, explain), pe3 is all seven, and any other set is pe: + features, e.g. pe:names+flat.

Finding behind labels

The prompts label several facts in words, not as the variable name with spaces. Nearly every test household has at least one: hours appear in 17 of 20, employer coverage and premiums in 11 each.

• Tool v2's names feature can't read these labels.
• Its notice says the facts carry the engine's variable names.
• A Flash-Lite smoke run fell back to employer_sponsored_insurance_premiums: 0 and weekly_hours_worked, and the SNAP work rules don't read the latter.

pe2 keeps that text because it ran with it; no other feature set says it.

Review round 1 (GPT-6.1 Sol, REQUEST_CHANGES, 7 findings): fixed in 6478e8f

1. Model arguments could switch on features their arm doesn't offer. The worker read __tool_features__ (and, as at 5bbd571, __tool_version__) from the arguments. Pools now send features in a tuple frame that no JSON can produce. Workers validate the frame, and such keys in a model's arguments are refused. No run record ever carried them.
2. A resume could mix treatments. Records now keep their tool features and calculator-schema hash, and a resume compares features, notice and schema.
3. Labels missed rendered labels. The index is now built from the prompt renderer for every promptable, labeled and reference-stated input. Dropped facts are ignored, and a label that names two inputs isn't read.
4. Person-level keys in flat. Keys other than name, role and facts are refused.
5. Flat merged facts before normalizing. Facts are now placed under the name written and compared once normalized. Roles set the relationship inputs, and a stated relationship that contradicts its role is refused.
6. Descriptions. They now promise only what each feature does.
7. Diagnostics. They read people by the flat format's default names.

Review round 2 (GPT-6.1 Sol, REQUEST_CHANGES, 2 minor findings): fixed in d570b8e

All seven round-1 findings were confirmed resolved.

1. A malformed repeat of a household fact was dropped silently. It now reports its validation problems, and only a repeat validated as equal is dropped quietly. A malformed first value is reported once, never as a conflict.
2. One harness failure blocked a custom-arm resume. The fallback record (harness_error_record) now carries the notice hash and treatment. The resume check (resume_treatment_problems) looked only at records the resume keeps, on the view that a rerunnable record is replaced under the current profile; round 3 showed it need not be.

Review round 3 (GPT-6.1 Sol, REQUEST_CHANGES, 1 major finding): fixed in 6b00a6f

Both round-2 findings were confirmed resolved.

1. A rerunnable record from another treatment could survive a resume and be scored. A budget stop, lost workers or a narrower --scenarios leave it in place, and score did no treatment check. Now treatment_problems checks every record in the folder, rerunnable ones included, and command_score refuses a folder that mixes treatments, rebuilding the expected notices on the board's engine.

One allowance at score, not at resume. Before 6e1578f, the tool notice and the calculate description always named policyengine-us 2.41.0, whatever engine ran. Test batch 0's frontier calculator arms (b6d835a) were told 2.41.0 while computing on 2.60.4, as every run manifest shows.

• A folder written wholly under that text is one treatment and scores with a note.
• A mix of the two texts is refused.
• A resume into such a folder is refused, since its new records would state the board's engine.

Across all 39 run folders (660 records, free batch 1 included), score refuses none; the six frontier calculator folders take the note.

Review round 4 (GPT-6.1 Sol): AGREE at 6b00a6f

The round-3 finding is resolved, and there are no new findings. Sol re-ran the round-3 counterexample: a resume now refuses it before scheduling, including with lost workers or a narrower --scenarios, and score refuses it before analysis.

Two qualifications on how the score allowance is described:

• It applies when some record fails on the board's engine and none fails on the reference engine.
• It checks the recorded treatment, not the commit a record came from. So a synthetic post-6e1578fc record written with the reference engine's text would pass.

Neither gives a mixed-treatment bypass. Sol also found stating_engine sound under nesting and exceptions, hashless legacy records still accepted, and the hard-constraint reading sound: the guards restrict orchestration and change no request or calculation.

Invariants

• pe, web+pe and pe2 behave exactly as at 5bbd571 (the commit the test batches ran). A Hypothesis differential ran 4,000 generated calls through versions 1 and 2. Requests, errors, notices, descriptions, schemas and 15 model profiles are identical.
• Flat equals the benchmark's own situation. On policyengine-us 2.60.4, 8 frozen-board households described flat, and 12 with each fact written as the v2-pilot prompt writes it, compute every board output exactly as Scenario.to_pe_household does.
• Each feature changes only its own behaviour. Per-feature unit tests; across all 95 valid feature sets, only pe2's notice makes the variable-name claim.
• Labels alone reproduce the reference from the prompt's facts. Each household is built from what its prompt shows, under the labels shown, with the labels feature alone. Every board output equals the reference situation's: 30 frozen households in the test suite; all 100 frozen and all 20 test-set households in a one-off run; with or without take-up inputs.
• Worker equivalence. pe and pe2 pool messages are read as 5bbd571's worker read them (2,000 generated). The intended exception is that forged metadata keys are refused.
• pe, web+pe and pe2 send the same texts as 5bbd571 on this board's stated engine (2.60.4). 2 schemas and 18 profiles are identical. A teeth check fails, as it must, when 5bbd571 is left on 2.41.0.
• One treatment per scored folder. Score refuses any folder whose records don't all match the arm's treatment on the board's engine, with one exception: a folder written wholly under the pre-6e1578fc text. A resume refuses any mismatched record.
• Mutants. 32 hand mutants of the features and the review fixes are all caught: 19 for the features and round 1, 5 for round 2, 8 for round 3.

Status

• Tests: 210 passed, 4 skipped (frozen-engine only) at 6478e8f
• Review round 1: REQUEST_CHANGES, 7 findings, all addressed in 6478e8f
• Review round 2 (GPT-6.1 Sol): REQUEST_CHANGES, 2 minor findings, both addressed in d570b8e (fb35d82 formats)
• 5bbd571 differential rerun at d570b8e: identical (4,000 calls, 2,000 pool messages)
• Full suite at fb35d82: ruff check and format clean; 214 passed, 4 skipped (gated on policyengine-us 2.41.0), 5 deselected (the round-2 selection)
• Review round 3 (GPT-6.1 Sol): REQUEST_CHANGES, 1 major finding, addressed in 6b00a6f
• 5bbd571 differential rerun at 6b00a6f: identical (4,000 calls, 2,000 pool messages); stated-engine comparison identical (18 profiles)
• Full suite at 6b00a6f: ruff check and format clean; 218 passed, 4 skipped (gated on policyengine-us 2.41.0), 5 deselected (the round-2 selection). CI runs only on PRs into main, so this local run is the suite evidence.
• Review round 4 (GPT-6.1 Sol): AGREE at 6b00a6f
• End-to-end Flash-Lite pe3 smoke run through real workers: 0 tool errors on 3 households (first smoke run: 7)
• Ablation pilot: 15 arms × 3 cheap models × 20 households, from a detached checkout of 6b00a6f (run_cheap_ablation.sh in the run folder). Batch 1, the free run and the review are done. It waits for the host: not before Route A's r3 window (route-a-release-d1811-r3) releases.
• Batch 1 and the free-plan run finished; merging into Run v2 on fresh Microcosm households in judged batches #240's branch

No published number changes; this is the unpublished tool pilot.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
policybench-site	Ready	Preview	Oct 24, 2026 5:31am UTC

[MaxGhenis]

Merged at the reviewed head 6b00a6f with a merge commit (5d807bb), so the reviewed commits stay reachable from #240's branch.

Reviews (GPT-6.1 Sol on Subfleet, review/hard):

• r1 REQUEST_CHANGES, fixed in 6478e8f
• r2 REQUEST_CHANGES, fixed in d570b8e
• r3 REQUEST_CHANGES, fixed in 6b00a6f
• r4 AGREE at 6b00a6f

Evidence: the full suite at 6b00a6f (218 passed, 4 skipped, ruff clean); the 5bbd571 differential (4,000 calls, 2,000 pool messages identical); the stated-engine comparison (18 profiles identical); the 39-folder treatment check (660 records, none refused by score). The ablation pilot waits for the host after Route A's r3 window.