[anth-volk]

Fixes #918

Merge order: step 2 of 3 of removing x and y from UK constituency and local-authority records. Both requirements are met: PolicyEngine/policyengine-app-v2#1436 is deployed, so the app reads no x or y, and #985 (step 1, fixes #984) is deployed, so the gateway accepts records without them. #924 (step 3, the contract removes them) follows once this is deployed and no breakdown started before this deploy is still running.

Same-Repository Draft PR Check

• This PR is a draft.
• This PR is from a branch in PolicyEngine/policyengine-sim-api, not a personal fork.

Summary

New results stop carrying x and y (each area's integer map position) on UK constituency and local-authority records, and the executor stops shipping coordinates_lad22.csv and coordinates_lad23.csv. Requests never reach apps built for older .py versions, so nothing is kept for them. This PR first also made x and y optional in the contract; that change is now #985, on main, and with main merged in (1ecb51d9) this PR's diff is only the executor change.

Three commits, each with one behavioural change, a fourth that rewords one sentence of the migration doc, and, after the merges of main, b54712ce, which drops the wording that kept x and y for older apps:

1. Accept UK area records without x and y. Now on main as Accept UK area records with or without x and y #985, which carries the same code with reworded comments and test names. x and y on ConstituencyImpactRecord and LocalAuthorityImpactRecord default to None and use pydantic's field-level exclude_if, so a record without them validates and every dump leaves the keys out instead of writing null. Records with integer x and y still validate and keep them. The contract's pydantic floor rises to 2.12, the first release with exclude_if; the contract and the four packages whose locks record its metadata are relocked, and only that specifier line changes in each lock. The gateway's golden OpenAPI no longer lists x and y as required.
2. Stop sending x and y on UK area records. The executor reads only the code and name columns of .py's constituency lookup, attaches only names to local authorities, and no longer refuses an area without a map position (the "missing lookup coordinates" check). The beta integration check of a local-area breakdown now requires records without x or y.
3. Ship local-authority code lists instead of coordinate files. codes_lad22.csv and codes_lad23.csv (one code column) replace the coordinate files, with exactly the same codes in the same order. UKLocalAuthorityResources carries codes per boundary version and names_for(...) replaces metadata_for(...). The Stage 12 static runtime file smoke reports lad22_codes and lad23_codes.

How the JSON loses the keys

• The executor returns output.model_dump(mode="json") from SimulationOutputBuilder.serialize and LocalAreaBreakdownBuilder.serialize; Stage 12 stores the same dump as its aggregate.
• The gateway validates that dict into JobStatusResponse and /jobs/{job_id} serializes it with response_model_exclude_none=True, which already drops null fields.
• With exclude_if (from Accept UK area records with or without x and y #985), the dump itself has no x or y, so neither the Modal result, the Stage 12 aggregate nor the gateway JSON carries the keys. The gateway tests check the response JSON for a breakdown and a national result whose records omit x and y, and for records that send them as null.

Boundary-version detection

detect_uk_local_authority_boundary_version reads the names file and the two hard-coded distinguishing code sets, which this PR doesn't change. Each version's full code list, which names_for and the "not part of the detected boundary version" check use, now comes from the code files. The load-time checks stay: the names must exactly cover both lists, and each version's distinguishing codes must match its list. A test pins each list to a SHA-256 of the codes the coordinate files listed (374 for LAD22, 361 for LAD23), and new tests check the loader refuses a repeated code, a list with another version's distinguishing code, and a list with x and y columns.

Deployment code and environment variables

The Modal images change content only: add_static_runtime_files copies the whole static_runtime_files directory, which now holds names.csv and the two code lists. src/modal/v2_static_runtime_files_smoke.py changes only the keys it reports. No environment variable is added, removed or read differently. For reference, the smoke's wiring is unchanged:

• MODAL_TOKEN_ID and MODAL_TOKEN_SECRET: GitHub Actions secrets, set in the Run image smokes step's env: in pr-image-smoke.yml.
• PE_UK_PRIVATE_HF_READ_TOKEN: GitHub Actions secret, set in the same step; .github/scripts/modal-image-smoke.sh writes it to the Modal secret pe-uk-private-hf-read-token as HUGGING_FACE_TOKEN, which the smoke function attaches through hf_secret (required_keys=["HUGGING_FACE_TOKEN"]).

Compatibility

Results the executor returns after this deploy carry neither key, which #985's contract accepts. Breakdowns that started before this deploy may still return records with x and y; the contract keeps accepting them until #924 removes the fields, which is why #924 waits until no such breakdown is still running.

Testing

Written red-green: for each commit the tests were written first and seen failing for the expected reason, then the code was written and both were committed together.

• Commit 1 (now Accept UK area records with or without x and y #985): contract tests failed on Field required for x and y and on a null x and y in the dump; the gateway returned 500 for records without x and y; the generated client did not read such a breakdown as a LocalAreaBreakdownOutput.
• Commit 2: executor tests failed on "missing lookup coordinates" and on x in the constituency and local-authority records; the breakdown check's new cases failed with DID NOT RAISE.
• Commit 3: tests failed because the resources had no codes or names_for, the package still held the coordinate files, and the smoke reported the coordinate keys.

Commands run (focused, locally):

• uv run --frozen pytest on test_macro_output_uk_geography.py, test_local_area_breakdown.py and test_gateway_models.py (contract); test_endpoints.py and test_openapi_golden.py (gateway); test_openapi.py (entry); test_simulation_output_geographic.py, test_simulation_output_builder.py, test_uk_local_authority_metadata.py, test_stage12_modal_app.py, test_smoke_app.py, test_stage12_runtime.py, test_stage12_output_planning.py, test_stage12_worker_validation.py and test_modal_bundle_image.py (executor).
• Golden OpenAPI regenerated with python -m policyengine_simulation_gateway.generate_openapi; the diff removes x and y from both records' required lists and updates the constituency description.
• ./scripts/generate-clients.sh, then uv run --frozen pytest tests/ -m "not requires_gcp and not beta_only" in policyengine-apis-integ: everything passed except test_ping, which needs a gateway running on localhost.
• make check (ruff and Pyright in every package), python -m scripts.rules pyright --sync projects/policyengine-simulation-executor, python scripts/run_quality_guards.py with GITHUB_BASE_REF=main, python -m scripts.rules render (no changes) and .github/scripts/check-uv-locks.sh.
• After merging main with Accept UK area records with or without x and y #985: test_generated_client_reads_area_records_with_x_and_y (apis-integ) passes against the regenerated client, ruff passes on the changed test file, and .github/scripts/check-changelog-fragment.sh passes.
• Not run locally: the full test suites, the Modal image smoke and the beta integration tests, which CI and the deployment run.

🤖 Generated with Claude Code