[anth-volk]

Fixes #973

Same-Repository Draft PR Check

• This PR is a draft.
• This PR is from a branch in PolicyEngine/policyengine-sim-api, not a personal fork.

Summary

After this deploys, the Simulation Entrypoint runs and stores whole economy reports on v2 without the old gateway, and a small staging report runs after every staging promotion. No user reaches it yet: the Public API's /v2/reports (a separate PR) is its first caller.

It builds on #951 (stacked on #950, #900 and #898, all on main) and merges two other unmerged PRs' branches with plain merge commits:

• Let the v2 report coordinator return a typed result to its caller #896 (the typed coordinator result and its pre-emption handling) in a915e342. main has split the modules Let the v2 report coordinator return a typed result to its caller #896 changes, so its changes are placed in main's modules: ReportResultDelivery in stage12_execution/comparison_runs.py; ReportSimulationCoordination and the Stage 12 coordinator in report_coordination.py; the child call cancellation and wait-failure report in coordinated_simulations.py; its tests in main's test support modules. Let the v2 report coordinator return a typed result to its caller #896 makes stage12_runtime/simulation.py 830 lines, so that merge also moves the simulation's record handling, unchanged, to stage12_runtime/simulation_records.py.
• Promote each staging deployment's tested Simulation Entrypoint revision #894 (staging promotes its tested revision) in b65541ee. Its test changes are placed in main's split deployment tests; the changelog keeps both entries, and the rules document is rendered again.

One Modal call per simulation run, one per comparison

• Contract (policyengine_simulation_contract.economy_runs): EconomySimulationCall names one simulation run, its policy's parameter values by name and period, its year, the PolicyEngine.py version, and the region and dataset its catalog version has (CatalogPopulation: the dataset's name, the household column and stored value that select the region, and each state's filter for a US national run). The call returns EconomySimulationOutput: its stored input and output files, the output plan's digest, row identity and bundle provenance. EconomyComparisonCall names a report run and both runs' policies and stored outputs (URI and SHA-256); it returns a SingleYearMacroOutput.
• Simulation calls (run_economy_simulation_us and _uk in the v2 Modal app): each checks that the call names the PolicyEngine.py version and a dataset the installed release has, plans the run's output columns (resolve_simulation_output_plan: the columns any report comparing it reads), and calculates it as a Stage 12 worker returning its result: no database rows, input and output stored under keys naming their SHA-256 in the Stage 12 artifact layout, a US national run in its 20 region-group segments. A simulation run has no report role; its Stage 12 input names it the baseline.
• Comparison calls (compare_economy_simulations, on the coordinator image): read both outputs, refuse one whose bytes differ from the digest sim recorded, require both to hold the same households and the columns the pair's comparison reads, compute the comparison with the Stage 12 comparison code, and return the result. They write nothing.
• Each top-level function reports its end: it wraps its work in reporting_call_end, which starts report_call_ended with the call ID and the run ID when the work returns or raises an Exception. A run ID that can be read from an otherwise malformed payload is still reported, so sim records the failure without waiting for the safety check. The unit tests check that each function is declared under the name sim starts it by; nothing checks a decorated function's __name__ at import, which the real Modal package does not allow.

Regions and datasets from the catalog

The entrypoint reads the run's region and dataset rows of its catalog version (CatalogReader, persistence library) and sends the dataset's name and the region's row filter. A simulation carrying a catalog population (options["catalog_population"]) loads that dataset and keeps the households its filter selects; the executor never calls policyengine.py's get_region or region registry for it. A stored value written as an integer (a state's FIPS number, a district's GEOID) is compared as one; a JSON list (England's nine regions) selects any of its values. A US national run's segments are scoped to their states' catalog filters, and every state the catalog filters is calculated. A region below the country without a household filter is refused. Stage 12 runs carry no catalog population and resolve their regions as before; no main-path code is used.

The Public API stores regions.region_type as the PostgreSQL enum v2_region_type, which has no comparison with text values, so the catalog reads compare it cast to text. Policy period dates are read in UTC. A policy value with no end date becomes a period key of its start date alone, which the worker reads as open-ended (a ParameterValue with no end date).

The tests' copies of the Public API's tables, checked in CI

The persistence tests create copies of the Public API tables sim reads and refers to, with only the columns sim uses. A copy that differs from the real table hides bugs: a varchar copy of regions.region_type let the comparison above, which PostgreSQL refuses, pass every test. So:

• tests/test_public_api_schema.py compares, for every table the copies create, each column's type (an enum by name and values), whether it allows null, and each primary key and unique constraint by name and columns, with a database the Public API's own v2 migration chain built. A column or table added to the copies is compared with no other change, so covering households and policies needs only their columns in tests/public_api_support.py.
• A new pr.yml job, Public API schema, checks out PolicyEngine/policyengine-api at its default branch, builds its tables with its v2 Alembic chain (.github/scripts/public-api-schema-reference.sh, the same commands and settings as the Public API's own Alembic v2 job), and runs that comparison, which fails rather than skips there. CI complete needs it. It follows the default branch rather than a pinned commit, so a change there to a column sim reads fails sim's next CI run instead of waiting for someone to move a pin; the cost is that such a change turns sim's CI red until the copies (and sim's code, if it reads the column) follow. The script prints the commit it migrated, and a failed comparison names it.
• Against the Public API's default branch the comparison found the copies differed, and they now match: parameter_values.value_json is jsonb (its baseline revision created it as json, a later one changed it); tax_benefit_model_versions.model_id and version, and policies.tax_benefit_model_version_id, refuse null; and the primary keys and the model name's unique constraint carry the Public API's names.
• The new rule sim.persistence.public-api-copies-match records this, with a proof: the copies with regions.region_type as varchar(64) are refused, and the current ones accepted. It changes the rules directory, so it needs a core developer's approval.

Runs, reuse and comparisons in the entrypoint

• EconomyReports (economy_reports; its requests, errors and store protocols are in economy_report_types) finds or creates the report's two economy simulations (so their runs can be reused), creates the report and its run on the catalog version its PolicyEngine.py version names, finds or creates a run of each simulation (reusing one that is unfinished or succeeded within 28 days), and starts the calls of the runs it created. A policy saved on another catalog version is applied by parameter name; a run whose policy changes a parameter its version lacks fails as invalid_input, naming the parameters. Catalog rows a call cannot carry are refused before anything is written. Once both runs have succeeded (at once when both were reused, or when a call's report is recorded) the comparison starts; outputs no longer offered fail it as result_expired.
• A submission starts only the runs it created. A reused run still pending is started by the submission that created it, which may be a labour supply report asking for more columns; if that submitter stopped, the safety check starts it after ten minutes for every report run comparing it, with the columns any of them needs.
• A forced start (a labour supply report's, or a forced re-run) always gets an attempt of its own: when it loses the insert of an attempt number to another caller, it adds the attempt after it.
• A report migrated from v1 without a run is calculated when it is first opened, through a system run. A system run returns the report's newest run when it has one, and RunStore.create_first_report_run records a first report run under a transaction-level advisory lock on the report, so simultaneous first openings share one run. A losing caller starts the runs it created that the shared run compares and leaves its others pending for the safety check.
• Finished calls are recorded with two new recorders (economy_recorders): a simulation run's input and output files are linked to it (output_files) in the transaction that marks it succeeded; a comparison's result is stored as the result tables' rows.
• Each safety-check pass also starts the comparisons whose runs have succeeded and the runs a submitter left pending for ten minutes, item by item, and fails a pending run no report run compares.

Routes

POST /v2/economy-reports, GET /v2/economy-reports/{report_id}, GET /v2/economy-reports/by-legacy-id/{legacy_report_id}, POST and GET /v2/economy-reports/{report_id}/runs, and GET /v2/simulations/{simulation_id} (economy_routes). Submissions need simulation:submit and reads simulation:read. A report reads with its newest run, the run's status (not_run for a migrated report without one), its stage while it runs (queued, simulating, outputs, from the run rows) and its error, and, once compared, the result rebuilt from its rows in today's format with UK area names from the catalog. A household report is not found on these routes.

Each route declares its error answers in the OpenAPI document, so the generated client parses them: 404 and 503 as EconomyErrorResponse (detail as text), 422 as EconomyRefusalResponse, whose detail is sim's reason as text or FastAPI's list of refused fields (EconomyRequestFieldError), and, on the two report reads, 500 as EconomyErrorResponse when a succeeded run's stored result cannot be rebuilt. Without the simulation database or the Modal environment the routes answer 503. The generated client gains submit_economy_report, get_economy_report, get_economy_report_by_legacy_id, run_economy_report, list_economy_report_runs and get_simulation; the comparison with the old gateway's contract leaves them out.

Cancellation and pre-emption

• A segmented US national run cancels every segment call it started when its call stops for any reason: a failed segment or dispatch (an Exception), and Modal's cancellation (InputCancellation) or pre-emption (KeyboardInterrupt), which Let the v2 report coordinator return a typed result to its caller #896's segment runner handles; the economy functions keep that behaviour, and a stopped call sends no report.
• Pre-emption is never a failure: the interrupted attempt sends no report, Modal runs the input again under the same call ID (which sim's run rows still hold), and the re-run stores its output under its own digest key and reports when it ends.
• Differs from the original design, following the pre-emption decision recorded in Let the v2 report coordinator return a typed result to its caller #896: a restarted call does not resume a claim or reattach to the calls its interrupted attempt started. It starts over, after its interrupted attempt cancelled them. There is no claim to resume in Modal (sim's run rows keep the call ID), and an identical artifact is reused because its key names its digest.

Staging check

After each staging promotion, the economy_staging_check job submits a report on Wales, current law compared with itself, to the stable URL. When the submission reused a run an earlier deployment finished, it runs the report again with a fresh run, so this deployment's functions calculate it (one simulation call and two comparison calls in that case, one of each otherwise). It waits up to twenty minutes and reads the result back. It needs a staging catalog version for the deployed PolicyEngine.py version. Its failure is shown in the run without holding back the production deployment (continue-on-error).

Deliberate limits, for the reviewer

• Report options must be empty. An SPM selection or cliff impacts change a simulation's own output, and a run's reuse key (simulation, catalog version, year) cannot tell which it was calculated with. The app's economy requests send neither today.
• Labour supply responses: a report's comparison reads labour supply columns from both simulations when either policy changes an elasticity. A run plans them only when asked, because in the UK they mean simulating again, so such a report always calculates both runs again and asks for them; other reports reuse runs.
• UK map positions are not in the region catalog, so a stored result is read back with the areas' names and null x and y.
• UK-wide and nation reports on a release that routes UK areas to a local-area dataset return no constituency or local-authority lists, as on the main path; the separate local-area breakdown run (LocalAreaBreakdownOutput) is not stored by the result tables yet.
• Time limits are unchanged: the US simulation call 3,900 s, the UK one 3,000 s, each segment call 3,600 s, the comparison call 1,800 s. A restart does not extend them, and no Modal call waits for the simulation calls: the entrypoint starts the comparison once it has recorded both runs.
• Overlap with Calculate and store households through the v2 household functions #957 (the household path, on the same base): both add routes, recorders and Modal functions to the same modules (app.py, simulation_database.py, run_dispatch.py, the v2 Modal app and its tests, the deployment workflow). Whichever merges second rebases. The OpenAPI changes here touch only the economy routes and add three schemas, none of which Calculate and store households through the v2 household functions #957 declares.

Environment variables

• SIMULATION_DATABASE_URL (entrypoint and safety-check job; read by the new routes, recorders and resumption): unchanged wiring from Give the Simulation Entrypoint its own simulation schema, migration chain and database connection #898 and Record simulation runs and report runs, and their outcomes when Modal reports a call ended #950. Provider: Secret Manager, named by the beta/prod variable SIMULATION_DATABASE_URL_SECRET_NAME, mounted by deploy-cloud-run-simulation-entry.sh, and copied at the pinned version into the safety-check job. Needs a new database grant (operator step): the runtime role must read public.tax_benefit_models, public.tax_benefit_model_versions, public.regions, public.datasets, public.parameters and public.parameter_values. Not verified in a deployment.
• MODAL_TOKEN_ID, MODAL_TOKEN_SECRET (entrypoint and safety-check job; now also used to start economy calls): unchanged wiring from Record simulation runs and report runs, and their outcomes when Modal reports a call ended #950. Provider: Secret Manager, named by MODAL_TOKEN_ID_SECRET_NAME and MODAL_TOKEN_SECRET_SECRET_NAME. Not verified in a deployment.
• SIMULATION_CALL_REPORT_MODAL_ENVIRONMENT (entrypoint: now also the Modal environment economy calls start in; safety-check job: newly copied from the serving revision). Provider: the reusable workflow input modal_environment, written by deploy-cloud-run-simulation-entry.sh as a plain variable; the safety-check deploy script now copies it with the other plain variables. Verified in the scripts and their tests; not in a deployment.
• APP_ENVIRONMENT (now also names the deployment in economy artifact paths): unchanged; the workflow input deployment_environment, copied into the job as before.
• SIMULATION_CALL_REPORT_MODAL_WORKSPACE_ID: unchanged from Record simulation runs and report runs, and their outcomes when Modal reports a call ended #950; the reports the economy functions send are checked against it. Its operator step (setting MODAL_WORKSPACE_ID) is still open.
• Modal functions: STAGE12_ARTIFACT_BUCKET and GOOGLE_APPLICATION_CREDENTIALS_JSON (artifact writes and reads) and HUGGING_FACE_TOKEN (datasets) reach the new functions through the same Modal secrets the Stage 12 workers use (stage12-evaluation-runtime, stage12-evaluation-gcp-credentials and the Hugging Face secret), attached as worker_secrets; MODAL_ENVIRONMENT labels telemetry (set by Modal, with a local fallback). report_call_ended gets SIMULATION_ENTRY_URL from the v2 deploy job, as in Record simulation runs and report runs, and their outcomes when Modal reports a call ended #950. The secrets' contents are not verified here.
• Public API schema job (new): no secret. SIMULATION_TEST_DATABASE_URL, PUBLIC_API_SCHEMA_DATABASE_URL, SIMULATION_REQUIRE_DATABASE_TESTS and SIMULATION_REQUIRE_PUBLIC_API_SCHEMA are literals in the job's env: (the service container's disposable credentials), and the service gets POSTGRES_USER, POSTGRES_PASSWORD and POSTGRES_DB as literals. The script sets UV_PYTHON=3.12, and V2_MIGRATION_DATABASE_URL and V2_ALEMBIC_DISPOSABLE_TEST for the Public API's Alembic command only, from its input; it writes PUBLIC_API_SCHEMA_COMMIT to GITHUB_ENV, which the runner provides. The checkout uses the automatic github.token under permissions: contents: read, with persist-credentials: false; the Public API's repository is public.
• Staging check job: SIMULATION_ENTRYPOINT_TEST_AUTH_ISSUER, _AUDIENCE, _CLIENT_ID and _CLIENT_SECRET from the beta environment secrets the deployed-auth checks already use (SIMULATION_ENTRYPOINT_AUTH_ISSUER, SIMULATION_ENTRYPOINT_AUTH_AUDIENCE, SIMULATION_ENTRYPOINT_TEST_AUTH_CLIENT_ID, SIMULATION_ENTRYPOINT_TEST_AUTH_CLIENT_SECRET), reaching the reusable workflow through secrets: inherit and the job's environment: beta; the stable URL and PolicyEngine.py version from the deploy_entrypoint and prepare job outputs. That the beta secrets exist is not verified here.

Waits for

• Store every simulation result as rows and rebuild today's result format from them #951, Record simulation runs and report runs, and their outcomes when Modal reports a call ended #950, Add simulation and report definitions to the simulation schema #900 and Give the Simulation Entrypoint its own simulation schema, migration chain and database connection #898 (this branch's base), and Let the v2 report coordinator return a typed result to its caller #896 and Promote each staging deployment's tested Simulation Entrypoint revision #894, whose branches it merges.
• Operator steps: sim's database roles and grants (including the new SELECT grant above), MODAL_WORKSPACE_ID and the safety-check schedule from Record simulation runs and report runs, and their outcomes when Modal reports a call ended #950, and a published staging and production catalog version for the deployed PolicyEngine.py version.
• The Public API's /v2/reports needs a published release of the simulation client containing these routes; publishing follows the production deployment.

Testing

On the head commit, locally:

• contract 396 passed; persistence 227 passed against PostgreSQL 17 (including simultaneous starts that wait on real row locks and simultaneous first report runs); Stage 12 persistence 40 passed; entrypoint 1,049 passed; executor 1,992 passed and 22 skipped; gateway 268 passed;
• the Public API schema job's steps, run locally against the Public API's default branch at 7de28683f: the migration and the comparison pass. With the copies as they were, the comparison listed 14 differences; with the old varchar(64) copy of regions.region_type, or a json copy of value_json, it fails on that column alone;
• ruff 0.15.22 (check and format), Pyright on every package, scripts/run_quality_guards.py (including the file-length rule) and the rules tool's tests;
• every commit on the branch was checked the same way for the packages it changes.

The OpenAPI document was generated from this branch and the Python client generated from it with openapi-python-client 0.29.0. Driven against the app with in-memory stores, the client parses each declared error answer: sim's own 422s and FastAPI's, 404 on every read, and 503 on every route.

An end-to-end run against a migrated PostgreSQL database with Modal replaced by a recording dispatcher submitted a report, recorded both simulation calls and the comparison, and read the result back rebuilt exactly from its rows; a re-run reused the runs.

Not run: anything against a deployed service, Modal app or database, including the staging check.

🤖 Generated with Claude Code