[anth-volk]

Fixes #993

Same-Repository Draft PR Check

• This PR is a draft.
• This PR is from a branch in PolicyEngine/policyengine-sim-api, not a personal fork.

Summary

A scheduled Modal function now marks the Public API's report runs and simulation runs failed when they are still running but the Modal call recorded for them has ended, for example because its container ran out of memory before it could record the outcome. It runs every ten minutes, never fails a run whose call is still running or whose state Modal cannot report, and two sweeps at once fail each run once.

This builds on #992, which records each run's Modal call ID when the coordinator or a worker claims the run; review only the commits after it.

How it tells that a call has ended

Every ten minutes the sweep lists the report runs, and then the simulation runs, that have been running under a recorded Modal call for at least ten minutes (counted from started_at, or updated_at when there is none). For each recorded call it asks Modal once for the call graph the call is part of (modal.FunctionCall.from_id(call_id).get_call_graph()) and reads the status of the inputs whose call ID is the recorded one, wherever they are in the graph: a worker spawned inside a coordinator's input is nested under it.

• any PENDING input: the call is still running, or waiting to run again; Modal restarts a pre-empted call under the same call ID and reports it pending meanwhile;
• every input SUCCESS, FAILURE, TERMINATED, TIMEOUT or INIT_FAILURE: the call has ended, and the run is marked failed with modal_call_ended: The Modal call recorded for this run ended without recording the run's outcome.;
• no input with that call ID, or any error asking (an unknown call, Modal unreachable): unknown, and the run is left as it is.

A simulation run is listed only while no report run that combines it is running. That report run's coordinator records its children's outcomes itself; after a pre-emption it cancels them and starts them again under new calls, and a sweep in that gap would otherwise fail a run that is about to start again. Report runs are swept first, so a dead coordinator's simulation runs are reached in the same sweep.

A run still running 24 hours after its first claim is failed whatever Modal reports, with or without a recorded call (run_exceeded_maximum_age). No attempt of these functions can run nearly that long (the coordinator's own time limit is 90 minutes), however often Modal pre-empts and restarts it; the limit stops a run staying running for good when Modal keeps reporting a call it won't retry as pending, no longer describes a call, or no call was recorded.

The failure is one conditional UPDATE … WHERE id = … AND status = 'running' AND modal_call_id = <the listed call>, so a run that finished, or that moved to a new call, since it was listed is left alone, and a second sweep running at the same time changes nothing the first changed.

Where it runs

A new Modal application, policyengine-simulation-run-sweep (src/modal/run_sweep_app.py), with one function, sweep_stuck_runs, on the cron schedule */10 * * * *, at most one container at a time, with the Modal worker role's database secret (policyengine-modal-worker-database) and the locked package group the v2 images install, without the bundle's data. It is not versioned with the .py bundle, so one copy runs per Modal environment rather than one per deployed v2 application. The deploy workflow doesn't deploy it yet; an operator deploys it (below).

Not covered: household jobs, which nothing records from Modal yet. A run marked running with no recorded call is failed only by the 24-hour limit.

Tests

The store's listing and conditional failure run against a disposable Postgres migrated with the Public API's v2 chain, as the Modal worker role, including two sweeps started together in threads, a run past the 24-hour limit with no recorded call, and simulation runs combined by pending, several or no report runs; they skip when REPORT_RUN_TEST_DATABASE_URL and REPORT_RUN_TEST_SEED_DATABASE_URL are unset, as in CI. The sweep's decisions and the call-graph reading use stand-ins for Modal and the store. The Modal application's declaration is checked with the repository's fake Modal module.

Environment variables

• MODAL_WORKER_DATABASE_URL: read by sweep_stuck_runs. Provider: the Modal secret policyengine-modal-worker-database, created by an operator in the staging and main Modal environments (the step Record report runs and simulation runs from the v2 Modal coordinator and workers #992 waits for). Wiring: modal.Secret.from_name(..., required_keys=["MODAL_WORKER_DATABASE_URL"]) attached through secrets= to sweep_stuck_runs. Not verified: the secret does not exist yet.
• The Modal client in the container authenticates as the workspace the app is deployed in; no other variable is read.

What this waits for before merging

1. Record report runs and simulation runs from the v2 Modal coordinator and workers #992 merged (this PR is stacked on it), with everything it waits for: Record each run of a v2 simulation and report, and key results by run policyengine-api#4162 deployed, the Modal worker role and its grants, and the Modal secret.
2. An approving review.

After merging, an operator deploys the app once per Modal environment, from projects/policyengine-simulation-executor:

uv run --frozen modal deploy --env=staging src/modal/run_sweep_app.py
uv run --frozen modal deploy --env=main src/modal/run_sweep_app.py

and checks that modal app list shows it and that its first scheduled run returned a summary (checked, failed, running, unknown, left, failed_for_age) without errors in modal app logs policyengine-simulation-run-sweep.

🤖 Generated with Claude Code