Record each run of a v2 simulation and report, and key results by run

.github/scripts/test_alembic_v2_lifecycle.sh

@@ -5,4 +5,5 @@ set -euo pipefail
 uv run --frozen --extra dev pytest -q \
   tests/unit/v2/test_alembic_v2.py \
   tests/integration/test_alembic_v2_lifecycle.py \
-  tests/integration/test_alembic_v2_catalog_tables.py
+  tests/integration/test_alembic_v2_catalog_tables.py \
+  tests/integration/test_alembic_v2_simulation_runs.py

.github/workflows/v2-integration-check.yml

@@ -82,8 +82,8 @@ jobs:
         run: uv run --frozen --extra dev coverage run -a --branch -m pytest -q tests/integration/test_v2_policy_persistence.py tests/integration/test_v1_policy_dual_write.py tests/integration/test_mysql_policy_dual_write.py tests/integration/test_v2_user_policy_mirroring.py tests/integration/test_v1_user_policy_dual_write.py
       - name: Test signed-in identity provisioning and user associations
         run: uv run --frozen --extra dev coverage run -a --branch -m pytest -q tests/integration/test_v2_user_identity_persistence.py
-      - name: Test comparison-run persistence and privilege reconciliation as the managed roles
-        run: uv run --frozen --extra dev coverage run -a --branch -m pytest -q tests/integration/test_v2_stage12_comparison_persistence.py tests/integration/test_v2_privilege_reconciliation.py
+      - name: Test comparison-run persistence, the simulation API's roles and privilege reconciliation as the managed roles
+        run: uv run --frozen --extra dev coverage run -a --branch -m pytest -q tests/integration/test_v2_stage12_comparison_persistence.py tests/integration/test_v2_simulation_role_privileges.py tests/integration/test_v2_privilege_reconciliation.py
       - name: Test v2 household persistence and immediate v1 copying
         run: uv run --frozen --extra dev coverage run -a --branch -m pytest -q tests/integration/test_v2_household_persistence.py tests/integration/test_v1_household_dual_write.py tests/integration/test_mysql_household_spm_dual_write.py
       - name: Test the row counts v1 ORM updates report on MySQL

changelog.d/v2-simulation-runs.changed.md

@@ -0,0 +1 @@
+API v2's database records each run of a simulation and of a report with the versions it used, keys every result to the run that produced it, and stores household calculation jobs for one household, one year and one policy. Its simulation tables no longer have foreign keys to the tables the Public API writes. Result tables gain the economy output's fields that had no column.

docs/engineering/skills/alembic-migrations.md

@@ -225,6 +225,8 @@ and a revision runs once per database, so it can't repair drift.
 | `policyengine_v2_runtime` | The Public API on Cloud Run |
 | `policyengine_v2_catalog_writer` | `scripts/publish_v2_metadata_catalog.py` |
 | `policyengine_stage12_<environment>` | The simulation API and its Modal worker, on the temporary comparison-run tables |
+| `policyengine_v2_simulation_service` | The simulation API on Cloud Run: creates simulations, reports, their runs and household jobs, and reads the policy, dynamic, household and catalog rows a run needs |
+| `policyengine_v2_modal_worker` | The simulation API's Modal workers: write their runs' and household jobs' status and results, and read the rows a run needs. Its `UPDATE` is table-level: the workers' code, not the database, limits it to their own run's status and result columns |
 
 Every v2 table has an entry for every role, so "no access" is a decision, not
 an omission. The migration role owns every table and isn't listed. Three rules
@@ -237,6 +239,9 @@ check the declarations:
   `policyengine_v2_runtime`, catalog publication as
   `policyengine_v2_catalog_writer`, and the comparison-run services as
   `policyengine_stage12_test`, holding only their declared privileges.
+  `tests/integration/test_v2_simulation_role_privileges.py` runs the
+  simulation API's writes as its two roles; their code lives in the simulation
+  API's repository.
 - `api.v2.deployed-privileges-match`: every deploy runs
   `scripts/reconcile_v2_privileges.py apply` right after the schema upgrade,
   then `verify`, before the catalog is seeded.
@@ -248,6 +253,13 @@ table is missing. The roles themselves are provisioned outside the
 repository; `--create-missing-roles` is accepted only for the disposable test
 database.
 
+A pull request may declare a role before an operator creates it by listing it
+in `ROLES_AWAITING_PROVISIONING` in `privileges.py`. While a listed role is
+missing, `apply` and `verify` skip it and name it in their output and the job
+summary instead of failing; once it exists, the next deploy grants and checks
+its privileges. Remove the role from the set once it exists in staging and
+production, so that its absence fails the deploy again.
+
 The declarations include write privileges for code paths production doesn't
 turn on: the dual-write copies into `legacy_policy_mappings` and
 `legacy_household_mappings`, and the writes behind the native `/v2/policies`
@@ -270,6 +282,7 @@ When a pull request adds or changes a v2 table, or a connector's use of one:
      tests/integration/test_v2_stage12_comparison_persistence.py \
      tests/integration/test_v2_metadata_routes.py \
      tests/integration/test_v2_catalog_publication.py \
+     tests/integration/test_v2_simulation_role_privileges.py \
      tests/integration/test_v2_privilege_reconciliation.py
    ```
 
@@ -453,3 +466,120 @@ schema.
 
 The dropped values are not recoverable. Adding a required column fails on a
 table that has rows, so the downgrade works only while both tables are empty.
+
+### API v2 simulation runs
+
+Revision `2ffd50f47a2b` (parent `c614b6bcccfc`) was autogenerated after these
+model changes:
+
+- `simulations` keeps only what to calculate. Its execution columns
+  (`status`, `error_message`, `started_at`, `completed_at`,
+  `household_result`) move to the new `simulation_runs`, one row per run with
+  the catalog version, country package version and PolicyEngine.py version it
+  used, and its Modal call ID; `output_dataset_id` is replaced by the run's
+  output file (`output_uri`, `output_sha256`).
+- The new `report_run_simulation_runs` names the simulation runs each report
+  run combines, as its baseline or its reform.
+- `poverty` and `inequality` belong to a simulation run (`simulation_run_id`);
+  the other result tables belong to a report run and name the two simulation
+  runs they compare (`baseline_simulation_run_id`, `reform_simulation_run_id`).
+  `aggregates` and `change_aggregates` are unchanged.
+- `report_runs` gains its Modal call ID and two JSON results that have no
+  table (`labor_supply_result`, `cliff_impact_result`).
+- `household_jobs` calculates one household (`household_id`) for one `year`
+  under one policy, and records its catalog and package versions and its
+  Modal call ID.
+
+It has two post-generation corrections, each commented in the revision:
+
+1. **Dialect:** `simulation_runs.status` reuses the `v2_simulation_status`
+   type the baseline created for `simulations.status`. The generated
+   `create_table` would create the type again and fail, so the column uses
+   `postgresql.ENUM(..., create_type=False)`.
+2. **Reversibility:** the downgrade drops `v2_simulation_role` after dropping
+   `report_run_simulation_runs`, because Alembic does not drop a table's enum
+   type with it.
+
+The upgrade adds required columns without server defaults to `household_jobs`
+and the result tables, so it needs them empty, as they are while no managed
+role holds a write privilege on them; on a table with rows it fails, and the
+deploy stops before anything changes. The downgrade restores required columns
+the same way, so it needs `simulations`, `household_jobs` and the result
+tables empty. The previous Public API revision keeps working at this revision:
+none of its roles can read or write a table the revision changes, except the
+catalog writer, which only counts the rows of `simulations`, `reports` and
+`report_runs`.
+`tests/integration/test_alembic_v2_simulation_runs.py` checks that the
+revision changes no other table, and the v2 lifecycle test covers the
+downgrade and re-upgrade.
+
+The name `simulation_runs` is also a v1 Cloud SQL table. The v2 metadata
+validation (`policyengine_api/data/v2/metadata_validation.py`) now treats it as
+a name the two databases share, as it does `simulations`.
+
+### API v2 foreign keys between the two services' tables
+
+Revision `7a83a87d029a` (parent `2ffd50f47a2b`) was autogenerated after the
+models stopped declaring foreign keys from the tables the simulation service
+and its Modal workers write (`simulations`, `reports`, `household_jobs`) to the
+tables the Public API writes (`users`, `policies`, `dynamics`, `households`,
+and the catalog's `tax_benefit_models`, `tax_benefit_model_versions`,
+`datasets` and `regions`). The 14 columns stay as plain UUIDs: the service that
+writes a row checks the rows it names. Foreign keys between tables of the same
+service stay. Deleting a Public API row no longer clears or blocks the rows
+that name it: in particular, deleting a user leaves that user's ID in
+`reports.user_id`, which no managed role may update; the owner of a saved
+report is the user association, which the deletion removes. It has no post-generation corrections. Its downgrade restores the
+keys, so it fails while a column names a row the referenced table lacks.
+`tests/unit/v2/test_simulation_run_models.py` checks that no foreign key or ORM
+relationship joins the two services' tables.
+
+### API v2 economy output fields
+
+Revision `7f43d2c7c3c6` (parent `7a83a87d029a`) was autogenerated after the
+models gained the fields of the simulation API's economy output that had no
+column: `report_runs.spm_provenance` (JSON) and `data_version`; the winner,
+loser and no-change shares of `congressional_district_impacts`; and the UK
+disclosure control fields (`disclosure_tier`, checked to be `reported` or
+`approximate`, and the four `_lower`/`_upper` interval bounds) of
+`constituency_impacts` and `local_authority_impacts`. The output's detailed
+budget names neither a program's entity nor whether it is a tax, so
+`program_statistics.entity` and `is_tax` become nullable, and its unique key
+treats a missing entity as equal (`NULLS NOT DISTINCT`). It has no
+post-generation corrections. The three share columns are required, so the
+upgrade needs `congressional_district_impacts` empty; the downgrade needs
+every `program_statistics` row to have an entity and a tax flag.
+
+### API v2 result row identity
+
+Revision `2e3de7c572ed` (parent `7f43d2c7c3c6`) was autogenerated after the
+models made each result row tell apart what it measures:
+
+- the comparison result tables' unique keys include the pair of simulation
+  runs (`baseline_simulation_run_id`, `reform_simulation_run_id`) after
+  `report_run_id`, because a budget-window report run compares one pair per
+  year;
+- `decile_impacts.decile_variable` names the variable that groups households
+  when it is not the income variable's own deciles (for example
+  `household_wealth_decile`), and joins the table's unique key;
+- `poverty.filter_group` names the group a filter selects (`child` for
+  `age`), and joins the table's unique key.
+
+Missing values in those keys count as equal (`NULLS NOT DISTINCT`, which
+PostgreSQL supports from version 15; `poverty` and `program_statistics` use it
+from the earlier revisions too). It has no
+post-generation corrections and needs no table empty; its downgrade fails
+while a report run holds comparison rows of more than one pair, or a run holds
+rows that differ only in a removed column.
+
+### API v2 comparisons
+
+Revision `dce0caa329cc` (parent `2e3de7c572ed`) was autogenerated after the
+models moved the parts of a comparison that have no result table
+(`labor_supply_result`, `cliff_impact_result`, `spm_provenance`,
+`data_version`) from `report_runs` to the new `report_run_comparisons`, one row
+per report run and compared pair of simulation runs, because a budget-window
+run compares one pair per year. It has no post-generation corrections. The
+four `report_runs` columns it drops were added by the earlier revisions of
+this change and never written; its downgrade drops the new table and its
+rows.

docs/generated/rules.md

@@ -244,9 +244,9 @@ flowchart LR
 | `api.migration.bundle-supported-by-simulation-api` | The installed PolicyEngine bundle is the latest one the simulation API has published, checked on every pull request and before each staging and production deploy. | mechanical | enforced | yes | CI job `ensure-policyengine-bundle-supported-by-simulation-api` | `pr.yml:ensure-policyengine-bundle-supported-by-simulation-api` | — |
 | `api.migration.contracts-consistent` | Migration route groups and workflow contracts are internally consistent, and their generated docs are current. | mechanical | enforced | yes | quality guard `migration-contracts` | `pr.yml:quality-guards` | **unproven** |
 | `api.migration.cross-database-integration` | Cross-database integration tests pass against real MySQL and Postgres. | mechanical | enforced | yes | CI job `v2-integration-check` | `pr.yml:v2-integration-check` | — |
-| `api.v2.deployed-privileges-match` | Every v2 deploy grants the managed roles each privilege policyengine_api/data/v2/privileges.py declares right after the schema upgrade, reports privileges it doesn't declare without revoking them, and stops before the catalog is seeded if a declared privilege is still missing. | mechanical | enforced | yes | CI job `seed`, `catalog-runtime-integration` | `seed-v2-database.yml:seed`, `v2-integration-check.yml:catalog-runtime-integration` | yes |
+| `api.v2.deployed-privileges-match` | Every v2 deploy grants the managed roles each privilege policyengine_api/data/v2/privileges.py declares right after the schema upgrade, reports privileges it doesn't declare without revoking them, and stops before the catalog is seeded if a declared privilege is still missing. A declared role listed in ROLES_AWAITING_PROVISIONING that doesn't exist yet is named in the job summary and skipped instead. | mechanical | enforced | yes | CI job `seed`, `catalog-runtime-integration` | `seed-v2-database.yml:seed`, `v2-integration-check.yml:catalog-runtime-integration` | yes |
 | `api.v2.runtime-privileges-declared` | Every v2 table has an entry for each managed database role in policyengine_api/data/v2/privileges.py, and every privilege a v2 connector module needs (from its reads, creates, updates or deletes module name and its row locks) is declared for the role that runs it. | mechanical | enforced | yes | pytest `tests/unit/v2/test_privilege_manifest.py` | `pr.yml:test` | yes |
-| `api.v2.runtime-privileges-tested-as-runtime-role` | The cross-database integration tests run the v2 user services and the parameter tree read as policyengine_v2_runtime, catalog publication as policyengine_v2_catalog_writer, and the comparison-run services as policyengine_stage12_<environment>, holding only their declared privileges, so a statement those privileges don't allow fails CI. | mechanical | enforced | yes | CI job `v2-integration-check`, `catalog-runtime-integration` | `pr.yml:v2-integration-check`, `v2-integration-check.yml:catalog-runtime-integration` | — |
+| `api.v2.runtime-privileges-tested-as-runtime-role` | The cross-database integration tests run the v2 user services and the parameter tree read as policyengine_v2_runtime, catalog publication as policyengine_v2_catalog_writer, the comparison-run services as policyengine_stage12_<environment>, and the simulation API's table writes as policyengine_v2_simulation_service and policyengine_v2_modal_worker, holding only their declared privileges, so a statement those privileges don't allow fails CI. | mechanical | enforced | yes | CI job `v2-integration-check`, `catalog-runtime-integration` | `pr.yml:v2-integration-check`, `v2-integration-check.yml:catalog-runtime-integration` | — |
 
 ```mermaid
 flowchart LR