[MaxGhenis]

Require audited evidence for a UK regional release served on an accepted model mismatch.

Invariants:

• National certification never widens. The national dataset name/hash, model coverage and core checks remain separate from the regional exception.
• An accepted area mismatch requires a reviewer reason with a visible character and audited evidence bound to the exact dense cut URI, original manifest bytes, artifact identity/SHA-256, the regional build model and core, and the exact target model package/version. Certification re-reads both models' surfaces, recomputes the comparison and refuses every dropped column. There is no dropped-column override.
• Free-text limitations, the dense cut's pre-fix VAT limitation and a generated limitation naming the defaulted inputs survive Pydantic validation and are hashed with the bundle.
• TRACE carries the full regional record as a JSON-LD JSON literal. An area run's execution receipt is uncertified, identifies its actual build model, and carries a summary of the exception bound to the record's digest; a national receipt carries none, because its certificate digest already covers the record.
• Re-certifying with --drop-regional-release removes the exception, regional manifest digest, area routes and dense artifact without changing the national certificate.

The audit is relative to the regional build model, as María ruled. scripts/audit_regional_input_surface.py compares the entity-table columns the H5 stores, in each period, with the build model and core the regional manifest records and with the installed target model, keeping the baseline of PolicyEngine/microcosm#1348's comparison:

• dropped: stored columns the build model defines and the target does not. These refuse.
• never modelled: stored columns neither model defines, such as Microcosm's area codes, build and source ids, clone indices and SPI donor fields. They are recorded, not refused.
• defaulted: target inputs the file does not store, after the target's input aliases, that were not inputs of the build model. Inputs absent under both models are not changes.

The build model's surface is read in a throwaway uv run --isolated --no-project --no-config --default-index https://pypi.org/simple --exclude-newer <build.built_at> --python 3.13 --with policyengine-uk==<build> --with policyengine-core==<core> environment, without the caller's UV_* settings other than cache, Python-install and connection ones, so the audit needs uv and PyPI. --isolated keeps uv from reusing an environment that already meets both pins, such as a .venv in the working directory. The probe reports the versions it imported; any other resolution refuses, as does a manifest that records no build model, core or build.built_at. The evidence moves to schema version 2 (version 1, or no version, refuses) and carries both models' full variable and input lists.

Certification does not trust those lists (from Vahid's review). It reads the target model in process, so certify-data must run where the target policyengine-uk is installed, as the audit does, and reads the build model again with uv. Only the stored columns rest on the auditor's word, bound to each H5's SHA-256, and the record says so: stored_columns: auditor_attested. It then recomputes all three lists, records never_modelled_columns in regional_certification, names the defaulted inputs in a generated limitation, and gives their counts in its warning. verify_regional_certification checks an area receipt's summary against the manifest the run used.

Required downstream refresh: update the dashboard's vendored tests/fixtures/w4/execution_receipt.v1.schema.json from this repository's regenerated schema.

Validation:

• Vahid's second review re-ran the four touched test files at 4931d18f (1,056 passed, 3 skipped) and the real verify_model_surfaces on 2.151.0. The round-2 changes (--isolated, the required build time, stored_columns) were not run locally while a long build ran on the producer's machine; this PR's CI runs them.
• Before that, the four touched test files (test_measure_regional_inputs.py, test_regional_input_evidence.py, test_certify_data_release.py, test_execution_receipt.py): 1,056 passed and 3 skipped. One skip is the rdflib JSON-LD case, which passed when run with rdflib. The build-model probe and the re-read surfaces are faked in tests, so CI installs nothing from PyPI; one test runs the real probe script against the installed model.
• scripts/generate_receipt_schemas.py --check, ruff format --check and ruff check (latest ruff, as CI) pass. Mypy reports nothing in the changed code.
• A real audit of dense cut microcosm-uk-2024-25-dense-20261019T000432Z-ee5c2dbe (built with policyengine-uk 2.132.6, core 3.33.3), reading column names only: 0 dropped and 83 never-modelled under each target; 0 new defaults against 2.132.6 itself, 16 against 2.147.2 (the 16 the docs list) and 37 against 2.151.0. These match María's figures, including the 123 inputs already defaulted under 2.132.6. With the pinned probe, the 2.147.2 audit is identical and took 24 s after the download.
• certify_data_release on a scratch copy of the bundle, with the 2.147.2 national cut microcosm-uk-2024-25-national-20261023T075039Z-a133f37a, the dense cut and its 2.147.2 evidence, in an environment with 2.147.2 installed, re-read both real models and recorded accepted_model_mismatch with no dropped columns, 83 never-modelled columns and 16 defaulted inputs, named in a generated limitation, in about 11 s. The same evidence with a stored build-model variable claimed as never modelled was refused. The area-receipt summary of that record is about 4 KB; the full record is about 110 KB.

🤖 Generated with Claude Code

[juaristi22]

My ruling on the accepted mismatch: keep the evidence, and fix what counts as dropped.

I like requiring evidence for this exception, and producing it is cheap. I dry-ran this PR's own scripts/audit_regional_input_surface.py (at c669bbd9) on the 19 Oct dense cut, microcosm-uk-2024-25-dense-20261019T000432Z-ee5c2dbe, with policyengine-uk 2.147.2 and 2.151.0 (core 3.33.3). The download took 34 s and each audit 17–23 s. But as written, the audit refuses every Microcosm dense cut, so this route can never be used:

• Both audits refused, on 83 "dropped" columns. None of the 83 is a policyengine-uk variable in 2.132.6 either, the version the dense cut was built with. They are Microcosm's own columns: area codes, build and source ids, clone indices and SPI donor fields. They include constituency_code_oa and la_code_oa, the columns src/policyengine/countries/uk/regions.py scopes areas by. The same audit would refuse this file against its own build model.
• Columns really lost since 2.132.6: none, under either 2.147.2 or 2.151.0.
• The defaulted list over-reports the same way. It lists 139 inputs under 2.147.2 and 160 under 2.151.0, but 123 of those already default under 2.132.6. New since the build: the 16 the docs list for 2.147.2, and 37 for 2.151.0. The extra 21 on 2.151.0 include the gambling-duty inputs, liquid and solid fuel consumption and bonus sacrifice, so area breakdowns of those measures would use defaults there. That belongs in the limitations if the route is used on 2.151.0.

The port dropped the baseline that PolicyEngine/microcosm#1348's comparison keeps: "Columns neither model defines, and inputs absent under both, are not changes."

Please:

1. Count as dropped only stored columns that the regional build model defines and the target model does not. Record never-modelled columns as information. This needs the build model's variables too; Let anyone build, sign and certify a Microcosm UK cut microcosm#1348 reads them from a throwaway uv run --with policyengine-uk==<version> environment.
2. Report defaulted inputs relative to the build model: target inputs the file doesn't store (after aliases) that weren't inputs of the build model.
3. Keep refusing real drops, with no override. There are none today.
4. Fix the red test: test__given_bundle_wrapper_empty_reason__then_certifier_refuses_it expects the certifier to refuse an empty reason, but bundle.py's parser now refuses it first.

With those changes the evidence takes about a minute to produce and can be scripted, which is fine for Budget day. Until the new national and dense pair on 2.151.0 lands, we accept the mismatch only to test the machinery.

[juaristi22]

Replying to the ruling: your four requests are pushed, with main merged in first (331fec1).

1. Dropped now counts only stored columns the build model defines and the target does not. Columns neither model defines are recorded as never_modelled_columns. The build model and core come from the regional manifest, and the audit reads them with uv run --no-project --with policyengine-uk==<build> --with policyengine-core==<core> (c1810b0).
2. Defaulted inputs are relative to the build model: target inputs the file doesn't store, after aliases, that weren't build inputs (c1810b0).
3. Drops still refuse, with no override. Certification recomputes all three lists from the evidence, which moves to schema version 2 (c1810b0).
4. The red test now expects bundle.py's SystemExit(2), with the certifier's "needs a reason" refusal as its context (89a8f76).

The docs carry your measurements (f14690e). Re-running the audit on the 19 Oct dense cut reproduces them: no drops and 83 never-modelled columns under 2.147.2 and 2.151.0, with 16 and 37 new defaults.

[vahid-ahmadi]

@juaristi22 @MaxGhenis round 1, at f14690ee1.

Verdict: the four requested changes are in and the national certificate does not widen, but the evidence is checked for consistency, not for truth, so an edited file can still hide a drop. No must-fix findings; three shoulds.

The four requests

1. Dropped vs never modelled. Done. regional_input_surface.py:161-162 counts as dropped only stored columns in the build model and not in the target; columns in neither go to never_modelled_columns. Certification recomputes the same (certification.py:563-582).
2. Defaulted relative to the build model. Done: (inputs - effective) - build_inputs at regional_input_surface.py:168, after the target's aliases.
3. Drops refuse, no override, schema v2. Done. The audit refuses at regional_input_surface.py:517-522 and certification at certification.py:584-588. No flag in bundle.py or certify_data_release.py lets a drop through. schema_version: Literal[2] (manifest.py:243) refuses v1, and a v1 file without the new fields also refuses (probed).
4. Red test. Done. tests/test_certify_data_release.py:1583-1638 expects SystemExit code 2 with the certifier's "needs a reason" error as its context, and checks the bundle is unchanged.

Invariants

• National never widens: the exception lives only in the regional try (certification.py:1453), the regional core check still runs (:1473), and test_regional_acceptance_never_widens_national_coverage covers it.
• Binding: URI, manifest sha256, model package and version (certification.py:489), build model and core against the manifest (:504), and each artifact's repo, path, revision and sha256 (:526). Wrong values for each refuse in the tests and in my probes, including evidence for 2.151.0 against a 2.147.2 target.
• Whitespace-only reasons (" ", "\t\n") refuse (certification.py:471).

Findings

1. should: certification trusts the declared surfaces, so editing the evidence is a dropped-column override. Certification recomputes the three lists from the lists in the file but never checks those lists against a model or the H5. I certified four hand-edited files, all accepted:

   • a column stored, defined by the build model and absent from the target, left out of build_model_variables: recorded as never modelled, 0 dropped;
   • the same column added to model_variables: 0 dropped;
   • the same column left out of columns: the sha256 still matches, so it passes;
   • the defaulted inputs added to build_model_inputs: defaulted_inputs comes out empty.

   So "no dropped-column override" holds only if the auditor is honest. Two cheap fixes: when the installed policyengine-uk equals model_version, rebuild the target surface and aliases with _model_surface and compare (this keeps model loading out of the normal path, as regional_input_surface.py:240 intends); and run _build_model_surface again at certification (4 s with a warm uv cache in my run). The stored columns cannot be checked without the H5, so the docs should say the auditor attests them. If you'd rather not add the checks, please soften the body and data-certification.md:132 ("recomputes all three lists") to say they are recomputed from the audited surfaces.

2. should: list the defaulted inputs as a limitation automatically. The 16 (2.147.2) and 37 (2.151.0) defaults reach regional_certification.defaulted_inputs, TRACE, receipts and the certify warning. They do not reach limitations unless the operator types them; the docs ask for it on 2.151.0 (data-certification.md:178-179) but nothing enforces it. Nothing warns at area-run time either. Area results for measures that read these inputs (SDLT purchase flags, salary and bonus sacrifice, PIP points, gambling duty, fuel consumption) use defaults. That is fine for testing the machinery, as you ruled, but a user reading only the limitations would not know. Appending one generated limitation that names them, as the VAT one is appended at certification.py:598, closes that.

3. should: every UK receipt now carries the whole evidence. receipt.py:3987 attaches regional_certification, including the full evidence, to every UK receipt, national runs included, and receipt.py:648 adds it to execution_identity_sha256. The four surface lists alone are about 86 KB on 2.132.6 (1,170 variables, 300 inputs, each counted twice), copied into every receipt while the bundle and TRACE already hold them. Certified national runs gain nothing from it, since certificate_sha256 (receipt.py:859) already digests the data_releases.uk record that contains it. Area runs have no certificate, so they do need it, but a digest of the record plus the summary fields (both versions, reason, the three lists, limitations) would do.

4. nit: the uv run step is safe but not fully reproducible. No injection: the command is an argv list with no shell, Version() rejects inner spaces, flags and markers (regional_input_surface.py:273-280), and the probe refuses any resolved version that is not exactly the pinned one (:338-343), so " 2.132.6\n" or "2.132.6+x" fail closed. But dependencies other than the two pins are unpinned, and UV_INDEX_URL, UV_EXTRA_INDEX_URL, UV_CONSTRAINT and UV_OVERRIDE pass through (:59-61). An --exclude-newer set to the manifest's build date would keep the probe working as numpy and pandas move on.

5. nit: a reason of only a zero-width space ("​") is accepted and stored, because str.strip() keeps it. RegionalCertification.accepted_reason (manifest.py:278) has no pattern, while audited_by requires \S. Giving it the same pattern would catch this.

6. nit: schema_version defaults to 2 (manifest.py:243), so a file without the field is read as v2. It is harmless because v2's fields are required, but making it required matches "version 1 refuses" literally.

What I ran

• Fresh clone and uv venv (Python 3.13, policyengine-uk 2.132.6, core 3.33.3). The four touched test files: 1,023 passed and 3 skipped, as in the body.
• Ten probes through certify_data_release with the PR's own fixtures: the four edits in finding 1 (all accepted), blank and zero-width reasons, v1 evidence, a missing schema_version, and a target-version mismatch.
• The real _build_model_surface("2.132.6", "3.33.3") through uv: it matches the installed 2.132.6 surface exactly (1,170 variables, 300 inputs).
• CI at f14690ee1: lint, mypy, changelog, docs, release inputs and smoke imports pass. The four Test jobs were still pending when I posted.
• Not run: the real audit on the dense cut (no download), or anything on 2.147.2 or 2.151.0, so the 83, 16 and 37 counts are yours, not re-measured. I did not read the regenerated receipt schema line by line or the TRACE JSON-LD test with rdflib.

🤖 Generated with Claude Code

[juaristi22]

Thanks, @vahid-ahmadi. All six are addressed in commits up to f0870aa:

1. Edited evidence. Certification now re-reads both models' surfaces. It reads the target in process, so certify-data must run with the target installed, as the audit does, and the build model through the same uv probe. Any list or alias that differs refuses, so each of your three surface edits refuses now, with a test for each (1a93f7e). The stored columns can't be re-read without the H5, so the docs say the auditor attests them, bound to the H5's SHA-256. On the real dense cut, certification re-read 2.147.2 and 2.132.6 in about 11 s, and refused the evidence once a stored build-model variable was claimed as never modelled.
2. Defaults as a limitation. Certification appends one generated limitation naming every defaulted input, as it does the VAT one, and adds none when nothing defaults (57c493c).
3. Receipt size. Only an area run's receipt carries regional_certification now. It holds a summary (both versions, the reason, the three lists and the limitations) and record_sha256, the digest of the whole record, evidence included. A certified receipt can't carry one, and verify_regional_certification checks the summary against the manifest the run used. For the dense cut's record, that is about 4 KB instead of 110 KB (cb0da7f).
4. Probe reproducibility. The probe now runs with --no-config --default-index https://pypi.org/simple --exclude-newer <build.built_at>, with the build time in UTC and rounded up to the second. It drops every UV_* setting except the cache, Python-install and connection ones (1a93f7e).
5. Zero-width reason. A \S pattern lets it through too: it matches U+200B, U+FEFF and U+2060. So the reason, the auditor and each limitation now need a printable non-space character, in the audit and in certification. accepted_reason also gets the \S pattern, for whitespace-only values (444982b).
6. schema_version is now required (444982b).

The docs and changelog are in f0870aa.

[vahid-ahmadi]

@juaristi22 @MaxGhenis round 2, at 4931d18f0.

Verdict: all six round 1 items are in. Certification now rejects every edit we made to the model lists. One part is still taken on the auditor's word: the list of stored columns. That is documented and bound to the H5's sha256, and I think it is fine for this PR. No musts. There are two shoulds and two nits below.

Round 1 items

1. Edited evidence: mostly fixed. verify_model_surfaces (regional_input_surface.py:423) reads the target's variables, inputs and aliases again in process, and the build model again through uv. Certification calls it before comparing anything (certification.py:595). Our edits again, through certify_data_release with the PR's fixtures, where the build model defines a stored column and the target does not:

   • the column left out of build_model_variables, so it reads as never modelled: refused;
   • the column added to model_variables: refused;
   • the defaulted inputs added to build_model_inputs: refused;
   • the column only in columns_by_period: refused (inconsistent periods);
   • the column left out of columns: accepted, with 0 dropped;
   • new since round 1: the two defaulted inputs added to columns, as if the H5 stored them: accepted, with defaulted_inputs: [] and no defaults limitation.

   I also ran the real path once, on policyengine-uk 2.151.0 with core 3.33.3: 1,271 variables, 335 inputs and 4 aliases. The uv read matched the in-process read exactly and took 2.3 s. Removing one name from model_variables, build_model_variables or build_model_inputs refused each time, and the error named the missing variable.

   So only the stored-column list (artifacts.*.columns) is still not recomputed. As the last two cases show, it decides both the drops and the defaults. The docs say this plainly (data-certification.md:141-142), and the evidence binds the list to the H5 sha256, so anyone with the file can check it. I accept that for this PR. See should 1 below for a cheap way to close it.

2. Defaults as a limitation: done. certification.py:639-644 appends one generated limitation naming the defaulted inputs, adds none when nothing defaults, and does not add it twice. All three cases are tested.

3. Receipt size: done. Only area runs carry regional_certification (receipt.py:4081-4085). It is a summary plus record_sha256 of the whole record (receipt.py:921), and a certified receipt that carries one is rejected (receipt.py:649). verify_regional_certification (receipt.py:4181) now does the check that verify_certificate used to do, which is the right split.

4. Probe reproducibility: done in the code, but see should 2. The probe adds --no-config, --default-index and --exclude-newer, and strips every UV_* setting except the cache and connection ones (regional_input_surface.py:356-377).

5. Zero-width reason: done, and our suggested fix was wrong. We said a \S pattern would catch "​". It would not: Python's \S matches U+200B, U+FEFF and U+2060, as you said. readable() (regional_input_surface.py:107) is the right fix, and it now covers the reason, the auditor and the limitations.

6. schema_version required: done (manifest.py:243). A file without it and a version 1 file both refuse.

Injection through the probe pins. None found. The command is an argv list with no shell. Both build versions pass Version() (regional_input_surface.py:304). The evidence's copies must equal the manifest's before they reach uv (certification.py:526). The cutoff is re-emitted by strftime in a fixed format from a parsed datetime (:313), so built_at text never reaches argv. Any resolved version other than the pin fails closed.

Findings

1. should: re-read the stored columns when the H5 is on disk. The operator has the H5s locally just after the audit. If certify-data took the same paths and re-read the column names with the audit's metadata reader, checking each sha256 first, it would close the last override route. When the files are absent, it would fall back to the auditor's word, as now. If you would rather not, please record the basis in the certified record (say, stored_columns: auditor_attested), so a reader of the bundle sees it without opening the docs.
2. should: the pins do nothing when the caller's environment already meets both. When I ran _build_model_surface("2.151.0", "3.33.3", exclude_newer="2026-01-01T00:00:00Z") from the clone, it succeeded, although 2.151.0 was published on 23 Oct. uv -v shows "Base environment satisfies requirements", and sys.prefix was the clone's .venv. No throwaway environment was made, and neither the cutoff nor the index applied. With --isolated added, the same call refused, as it should. I found no wrong acceptance from this. The caller's environment only meets both pins when its policyengine-uk is the build version, and certification requires the target. But the second read is then not independent of the operator's environment, which may be another .venv in the working directory or an editable checkout reporting that version. The docs' "as of the manifest's build time" claim (data-certification.md:127-131) also fails. Adding --isolated to the argv at regional_input_surface.py:353-370 fixes it. Our round 1 nit suggested --exclude-newer without saying it needs this too.
3. nit: regional_build_cutoff returns None when the manifest has no build.built_at, and the probe then runs unpinned without saying so (regional_input_surface.py:313). regional_build_versions refuses on missing build versions. Refusing here too would be consistent, or the docs could say the pin is optional.
4. nit: the comment at regional_input_surface.py:269 ("Keep country-model loading out of certification's evidence-validation path") is now stale, because certification loads the target model on purpose.

What I ran

• Fresh clone at 4931d18f0, fresh uv venv (Python 3.13, policyengine-uk 2.151.0, core 3.33.3). The four touched test files: 1,056 passed and 3 skipped.
• Six hand-edited evidence probes through certify_data_release with the PR's fixtures, listed under item 1.
• The real verify_model_surfaces on 2.151.0 / 3.33.3, honest and with three edits, and the cutoff checks with and without --isolated. For the cutoff I used the real dense manifest's built_at, with the HF read-only login.
• CI at 4931d18f0: all 13 checks pass, including the four Test jobs.
• Not run: the real audit or certification on the dense cut with 2.147.2 as the build model, so your 11 s figure there is yours, not re-measured.

🤖 Generated with Claude Code

[juaristi22]

Thanks, @vahid-ahmadi. Round 2 is addressed at 517396a:

1. Stored columns. This PR keeps the auditor's word, as you allowed, and the record now states the basis: regional_certification.stored_columns: auditor_attested. The area receipt's summary carries it too (55011f6). Re-reading the H5s at certification would need certify-data to take the artifact paths; that could follow in another PR.
2. --isolated. The probe now runs uv run --isolated ..., so it always builds a fresh environment and the index and cutoff apply (55011f6). Thanks for catching that it could reuse a .venv that met both pins.
3. A missing build.built_at now refuses, in the audit and at certification, as missing build versions do (55011f6).
4. The stale comment on the lazy policyengine_uk import is rewritten (55011f6).

The docs and changelog are in 517396a. These changes weren't run locally, because a long build is running on my machine; this PR's CI runs them on the head.