From df612c4d24524d3913f5f7bbd4b5b599c66f055e Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 3 Nov 2026 02:53:00 +0000 Subject: [PATCH 1/5] Correct the CI timing comments and test notes after #253 Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01LwQq6FeG5zNPJXgBp8j1qj --- .github/workflows/ci.yml | 13 +++++++------ CLAUDE.md | 6 +++--- 2 files changed, 10 insertions(+), 9 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 959fbe5f..65a77ed7 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -31,10 +31,11 @@ jobs: test: runs-on: ubuntu-latest # Serially, the test job on main took 29.5 minutes at b0c8f2e and hit - # this limit at ec5413c (2026-11-02). In parallel it took 12.3 minutes - # with -n logical (4 workers, run 41526298328), and 12.5 to 16.2 minutes - # in three attempts with -n auto (run 41490222217). Fail a hung job - # instead of holding a shared runner for GitHub's 6-hour default. + # this limit at ec5413c (2026-11-02). In parallel it took 12.3 and 15.7 + # minutes with -n logical (4 workers, runs 41526298328 and 41529016470), + # and 12.5 to 16.2 minutes in three attempts with -n auto (run + # 41490222217). Fail a hung job instead of holding a shared runner for + # GitHub's 6-hour default. timeout-minutes: 30 steps: - uses: actions/checkout@v6 @@ -49,8 +50,8 @@ jobs: - run: uv sync --locked --extra dev # One worker per logical CPU (-n auto would count physical cores, since # psutil is installed); the "created: N/N workers" line gives the count. - # --dist loadfile keeps each test file on one worker; on the 4-core VM it - # ran faster than --dist load (10.1 minutes). + # --dist loadfile keeps each test file on one worker; on a 4-vCPU VM it + # ran faster than --dist load (9.2 to 9.7 minutes, against 10.1). - run: uv run pytest -m "not slow" -n logical --dist loadfile --tb=short app: diff --git a/CLAUDE.md b/CLAUDE.md index e33cbcc5..5bb9b2d3 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -3,7 +3,7 @@ ## Quick start ```bash pip install -e ".[dev]" -pytest # Run tests (all external calls mocked) +pytest # Run tests (see "Testing" for the one download) ruff check . # Lint ruff format . # Format ``` @@ -26,9 +26,9 @@ ruff format . # Format - `policybench/analysis.py` — Metrics and reporting ## Testing -- All tests mock external calls (LiteLLM, PE-US API) +- All tests mock external calls (LiteLLM, PE-US API), except the `tests/test_scenarios.py` tests that download the certified US dataset from Hugging Face - `pytest -m "not slow"` to skip slow tests -- `pytest -m "not slow" -n 4 --dist loadfile` to split them across four workers (CI starts one per CPU) +- `pytest -m "not slow" -n 4 --dist loadfile` to run the same tests on four workers (CI starts one per logical CPU) - Some tests read git history, so run the suite from a full clone (`git fetch --unshallow`) - Full benchmark runs are manual and expensive From 9312078b29ef0e94d1dad780348beb1fee0c2a0e Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Mon, 2 Nov 2026 22:29:44 -0500 Subject: [PATCH 2/5] Compare like runs in the CI timeout comment and fix the mocked-call list The timeout comment compared main's serial test job at b0c8f2e with parallel runs on merges with f4c3c38. It now gives each base's serial time beside its parallel runs: 29.5 minutes at b0c8f2e against 12.5 to 16.2 with -n auto, and 22.3 at f4c3c38 against 12.3 and 15.7 with -n logical. CLAUDE.md listed a PolicyEngine API among the mocked calls, though no test calls one. The list now names LiteLLM, provider batch adapters and gh, and says tests that need policyengine-us run it in-process. Co-Authored-By: Claude Opus 5.5 --- .github/workflows/ci.yml | 11 ++++++----- CLAUDE.md | 2 +- 2 files changed, 7 insertions(+), 6 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 65a77ed7..ca8e5eb4 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -30,11 +30,12 @@ jobs: test: runs-on: ubuntu-latest - # Serially, the test job on main took 29.5 minutes at b0c8f2e and hit - # this limit at ec5413c (2026-11-02). In parallel it took 12.3 and 15.7 - # minutes with -n logical (4 workers, runs 41526298328 and 41529016470), - # and 12.5 to 16.2 minutes in three attempts with -n auto (run - # 41490222217). Fail a hung job instead of holding a shared runner for + # Serially, the test job on main took 29.5 minutes at b0c8f2e, hit this + # limit at ec5413c and took 22.3 minutes at f4c3c38 (2026-11-02). In + # parallel it took 12.5 to 16.2 minutes in three attempts with -n auto on + # a merge with b0c8f2e (run 41490222217). With -n logical (4 workers) on + # merges with f4c3c38 it took 12.3 and 15.7 minutes (runs 41526298328 and + # 41529016470). Fail a hung job instead of holding a shared runner for # GitHub's 6-hour default. timeout-minutes: 30 steps: diff --git a/CLAUDE.md b/CLAUDE.md index 5bb9b2d3..c837bb18 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -26,7 +26,7 @@ ruff format . # Format - `policybench/analysis.py` — Metrics and reporting ## Testing -- All tests mock external calls (LiteLLM, PE-US API), except the `tests/test_scenarios.py` tests that download the certified US dataset from Hugging Face +- All tests mock external calls (LiteLLM, provider batch adapters, `gh`), except the `tests/test_scenarios.py` tests that download the certified US dataset from Hugging Face. Tests that need policyengine-us run it in-process. - `pytest -m "not slow"` to skip slow tests - `pytest -m "not slow" -n 4 --dist loadfile` to run the same tests on four workers (CI starts one per logical CPU) - Some tests read git history, so run the suite from a full clone (`git fetch --unshallow`) From 3f19548146a474a34e3a4e2599aa84fb56493403 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Mon, 2 Nov 2026 22:37:55 -0500 Subject: [PATCH 3/5] Name LiteLLM's model cost map fetch in the test notes LiteLLM fetches its model cost map from GitHub when a pytest process imports it (litellm/__init__.py and get_model_cost_map at v1.95.0), so the tests reach the network for more than the Hugging Face download. CLAUDE.md now names both calls and the variable that skips LiteLLM's fetch, and the quick start points to network use instead of one download. Co-Authored-By: Claude Opus 5.5 --- CLAUDE.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index c837bb18..93dcd464 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -3,7 +3,7 @@ ## Quick start ```bash pip install -e ".[dev]" -pytest # Run tests (see "Testing" for the one download) +pytest # Run tests (see "Testing" for network use) ruff check . # Lint ruff format . # Format ``` @@ -26,7 +26,9 @@ ruff format . # Format - `policybench/analysis.py` — Metrics and reporting ## Testing -- All tests mock external calls (LiteLLM, provider batch adapters, `gh`), except the `tests/test_scenarios.py` tests that download the certified US dataset from Hugging Face. Tests that need policyengine-us run it in-process. +- Tests mock LiteLLM's model calls, the provider batch adapters and `gh`, and the tests that need policyengine-us run it in-process. +- LiteLLM fetches its model cost map from GitHub when a pytest process imports it, and falls back to its bundled copy if the fetch fails. Set `LITELLM_LOCAL_MODEL_COST_MAP=True` to skip the fetch. +- The `tests/test_scenarios.py` tests download the certified US dataset from Hugging Face. - `pytest -m "not slow"` to skip slow tests - `pytest -m "not slow" -n 4 --dist loadfile` to run the same tests on four workers (CI starts one per logical CPU) - Some tests read git history, so run the suite from a full clone (`git fetch --unshallow`) From d978ddcd2235ae3e786073ecd6b8fb8b7ed00e42 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Mon, 2 Nov 2026 22:47:58 -0500 Subject: [PATCH 4/5] Name every process that fetches LiteLLM's cost map The policybench.cli runs that some slow tests start import LiteLLM too, so they fetch the cost map as well. Only some tests in tests/test_scenarios.py download the certified US dataset. Co-Authored-By: Claude Opus 5.5 --- CLAUDE.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 93dcd464..70a1838a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -27,8 +27,8 @@ ruff format . # Format ## Testing - Tests mock LiteLLM's model calls, the provider batch adapters and `gh`, and the tests that need policyengine-us run it in-process. -- LiteLLM fetches its model cost map from GitHub when a pytest process imports it, and falls back to its bundled copy if the fetch fails. Set `LITELLM_LOCAL_MODEL_COST_MAP=True` to skip the fetch. -- The `tests/test_scenarios.py` tests download the certified US dataset from Hugging Face. +- LiteLLM fetches its model cost map from GitHub in every process that imports it, and falls back to its bundled copy if the fetch fails. Each pytest process imports it, and so do the `policybench.cli` runs that some slow tests start. Set `LITELLM_LOCAL_MODEL_COST_MAP=True` to skip the fetch. +- Some `tests/test_scenarios.py` tests download the certified US dataset from Hugging Face. - `pytest -m "not slow"` to skip slow tests - `pytest -m "not slow" -n 4 --dist loadfile` to run the same tests on four workers (CI starts one per logical CPU) - Some tests read git history, so run the suite from a full clone (`git fetch --unshallow`) From 1e2a4269ba68993a34d177b55813ed93445465ac Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Mon, 2 Nov 2026 22:50:39 -0500 Subject: [PATCH 5/5] Say which pytest processes import LiteLLM Under pytest-xdist the controller collects no tests, so it never imports LiteLLM; each worker does, as does a serial run's one process. Co-Authored-By: Claude Opus 5.5 --- CLAUDE.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CLAUDE.md b/CLAUDE.md index 70a1838a..364cf817 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -27,7 +27,7 @@ ruff format . # Format ## Testing - Tests mock LiteLLM's model calls, the provider batch adapters and `gh`, and the tests that need policyengine-us run it in-process. -- LiteLLM fetches its model cost map from GitHub in every process that imports it, and falls back to its bundled copy if the fetch fails. Each pytest process imports it, and so do the `policybench.cli` runs that some slow tests start. Set `LITELLM_LOCAL_MODEL_COST_MAP=True` to skip the fetch. +- LiteLLM fetches its model cost map from GitHub in every process that imports it, and falls back to its bundled copy if the fetch fails. Each pytest process that runs tests imports it, and so do the `policybench.cli` runs that some slow tests start. Set `LITELLM_LOCAL_MODEL_COST_MAP=True` to skip the fetch. - Some `tests/test_scenarios.py` tests download the certified US dataset from Hugging Face. - `pytest -m "not slow"` to skip slow tests - `pytest -m "not slow" -n 4 --dist loadfile` to run the same tests on four workers (CI starts one per logical CPU)