[anth-volk]

Fixes #4199

Based on master. It was stacked on #4196, which merged. #4196's v2-integration-check.yml keeps the integration tests' coverage report as the integration-coverage artifact that this job downloads. master was merged in at 8ab6df648, the first run of the full pull-request workflow, and so of the diff-coverage job itself, on this pull request.

Summary

• New job diff-coverage in pr.yml:
  • It needs test and v2-integration-check.
  • It downloads every test shard's coverage.xml (test-coverage-shard-*) and the integration tests' coverage-v2.xml (integration-coverage).
  • It runs uv run --frozen --extra dev diff-cover "$RUNNER_TEMP"/coverage/*/coverage*.xml --compare-branch=HEAD^1 --fail-under=80.
  • CI checks out the merge commit GitHub builds for the pull request, so HEAD^1 is the commit that merge is built on: the base branch, or for a pull request stacked on another, the lower one's merge commit. For example, refs/pull/4200/merge (c0882ac1) is built on refs/pull/4196/merge (5ff9f725). The job runs only on the lowest pull request of a stack. Either way the diff holds only the pull request's changes. The event's base.sha isn't used; see "Why HEAD^1" below.
  • diff-cover counts a changed line as covered if any report covers it, and lists each file's uncovered lines in the job log.
• CI complete needs the job, so a pull request with less than 80% of its changed lines in policyengine_api/ covered can't merge. No ruleset change is needed.
• diff-cover>=10.6,<11 is added to the dev extra and locked with uv 0.13.4.
  • It locks diff-cover 10.6.0 (released 2026-09-22) and its new dependency chardet 7.7.0 (released 2026-10-18).
  • diff-cover 11.0.0 and 11.0.1 are less than two weeks old; moving to them is a separate change.
• New enforced metric rule api.quality.diff-coverage.
  • Its proof, tests/unit/test_diff_coverage.py::test_rule_proof_diff_coverage, commits a module, adds 1,000 lines, and runs the real diff-cover with --fail-under=80. It must reject 799 covered lines (79.9%, which diff-cover truncates to 79%) and accept 800 (80%).
  • The rule, like diff-cover, counts the changed lines coverage.py measures: the first line of each statement. A line that ran counts as covered even if one of its branches didn't.
  • Two tests run the job's own command from pr.yml with the locked diff-cover on a merge commit. One checks that master's changes after the branch left it aren't counted, with the event's base.sha at that older commit. The other checks that a line counts as covered if any report covers it, with one shard report and one integration report.
  • One runs coverage.py with the repository's .coveragerc and no imports, and checks that every module under policyengine_api/ is reported.
  • One measures a module with the repository's .coveragerc in one directory, then runs the job's command in another checkout of the same change, and checks that the change is measured.
  • test_only_lines_that_can_run_are_counted checks that .coveragerc leaves if TYPE_CHECKING: imports, a Protocol method's ... body and three spellings of pragma: no cover out of the missed lines. It also checks that an untested function whose comment mentions if TYPE_CHECKING: still counts.
  • Others check the job's wiring, that the rule, its proof and Codecov use the same minimum, and that diff-cover is locked.
• policyengine_api/libs/__init__.py and policyengine_api/routes/__init__.py are new.
  • The locked coverage.py 6.5.0 reports a module no test imports only inside a package. These two directories had no __init__.py, so their untested modules were in no report, and diff-cover would pass a new untested module there.
  • CI run 42041987958's integration report listed only one of the 15 modules in these two directories, libs/user_auth.py, which an integration test imports. Five of the six shard reports left out libs/simulation_api.py and libs/simulation_api_modal.py.
  • pyproject.toml's ruff namespace-packages comment no longer says they have none.
• .coveragerc changes:
  • It excludes if TYPE_CHECKING: blocks, which never run. Ruff's TC rules move imports there, so a fully tested change that adds type-only imports would otherwise fall below 80%. 192 modules under policyengine_api/ have such a block. The pattern matches only the whole statement: coverage.py searches each line of text, and a matching line that opens a block excludes the block, so the plain text would also hide an untested function whose def line has a comment mentioning it.
  • It excludes ... alone on a line: the bodies of Protocol methods and other stubs, which never run. In CI run 42046754813, 89 such statements were measured and none ran. Over the 346 modules, removing them is the only change this pattern makes.
  • Setting exclude_lines replaces coverage.py's default, so its first entry is the default pattern itself, \#\s*(pragma|PRAGMA)[:\s]?\s*(no|NO)\s*(cover|COVER). It is written with \# because a value starting with # is read as a comment.
  • It sets relative_files = True. Reports then carry <source>policyengine_api</source> and file names relative to it, instead of the runner's absolute checkout path. diff-cover looks a changed file up by its absolute path or its path from the repository root. A report written in another checkout directory, such as a container: job's, would otherwise match no changed file, and the job would pass with "No lines with coverage information".
• codecov.yml's patch status now shows the same target: 80%, still informational, since this job enforces the minimum.
• docs/engineering/skills/testing.md has a "Changed lines" section, which the rule cites, with the command to check locally after make test.

Why HEAD^1

The pull_request event's base.sha often stays at the commit where the branch left master, while the checkout merges into master as it is now. In 17 of 88 recent pr.yml runs the two differed, and in each of those base.sha was the older commit. Comparing with it counts every change that reached master in between as the pull request's own. The master ruleset doesn't require branches to be up to date, so a pull request can merge on that result.

Run 42039645804 (#4191) checked out 8b441b8a merged into 4f2960b6, with base.sha 22bb5e4d. Between those two master commits, master added 1,077 lines under policyengine_api/. That run predates the integration artifact, so it has six coverage reports, one per test shard. On those six, with their checkout path rewritten to a local copy of the merge commit, the locked diff-cover measured:

--compare-branch	Measured lines	Coverage
22bb5e4d (base.sha)	477	96%
HEAD^1 (4f2960b6)	39	97%

The 39 lines are #4191's own changes.

In CI, the test shards, the integration tests and this job all check out the same commit at the same path, so the reports' paths match without rewriting. The decision behind the 80% minimum is in tracker task 6.4 · 6: of the last 40 merged pull requests, 13 changed measured code, and only #4179 (75%) would have failed.

Stacked pull requests and the skip

The job has needs and no condition of its own, as tests/unit/test_ci_complete_gate.py requires. On a pull request above the lowest one in a stack, test and v2-integration-check are skipped, so diff-coverage is skipped too. CI complete fails there, as it already does for the other skipped jobs. On the lowest pull request, it runs once both have passed. If either fails, it is skipped and CI complete fails with them.

Design choices for review

• The job installs the whole dev environment (uv run --frozen --extra dev), including the country models, because the decision put diff-cover in the dev extra. A dependency group of its own (uv run --frozen --only-group, as telemetry-error-issues does) would install only diff-cover. That would make the job faster but leave diff-cover out of make install. Say if you prefer it.
• Only lines under policyengine_api/ count, because .coveragerc measures nothing else. A pull request that changes only tests, workflows or docs passes.
• What the check can't stop. A pull request can still leave its own lines out of the count by adding # pragma: no cover or a pattern to .coveragerc's exclude_lines, or change the job or its --fail-under value along with the test that pins it. The master ruleset doesn't require code-owner review, so none of these files needs a particular reviewer. Code that runs only in the Alembic checks, the environment-variable tests, push.yml's live tests or a subprocess counts as uncovered. Moving code into a new file counts its lines as changed, unless git pairs the new file with a deleted one as a rename (at least 50% similar), when only the edited lines count. The testing skill documents these.
• Another test now imports more modules. tests/unit/test_runtime_evaluated_annotations.py walks the packages with pkgutil.walk_packages, which skipped libs/ and routes/ while they had no __init__.py. It now imports their 15 modules too. Its 5 tests passed by node ID in a review run.
• __init__.py files rather than a coverage.py upgrade. coverage.py 7 has [report] include_namespace_packages, which would report these modules without the two files. Nothing holds coverage.py at 6.5.0; it is simply the locked version. Upgrading it would change how every CI test run is measured, so I kept that separate. The test that every module is reported fails for any future directory under policyengine_api/ without an __init__.py. Ruff's INP001 (implicit namespace package) can't catch that, because pyproject.toml declares all of policyengine_api a namespace package.

CI wiring and environment variables

• No step-level environment variable: the base is HEAD^1, from the checkout itself.
• RUNNER_TEMP: the runner's default variable, where the two download steps write (${{ runner.temp }}/coverage).
• UV_PYTHON: the literal "3.12" in the job's env:, as the other jobs that run uv set it.
• The artifacts come from earlier jobs in the same workflow run (test and the reusable v2-integration-check), and downloading them needs no secret.
• The job reads no secret. It inherits the workflow-level env: and uses neither variable there: ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION (the literal true) and OLD_SIMULATION_GATEWAY_URL (the repository Actions variable of that name, which exists). The job doesn't authenticate to GCP or Codecov.
• actions/checkout and astral-sh/setup-uv use the automatic github.token with the job's contents: read; the downloads use the runner's token for the same run's artifacts.
• actions/checkout@v4 with fetch-depth: 0 gives the merge commit's first parent for diff-cover's git diff HEAD^1...HEAD; a depth-1 clone lacks it.
• The job has a 15-minute timeout and no path filter or condition.

Testing

Red, at the first commit (tests and fixtures only):

UV_PYTHON=3.12 uv run --frozen --extra dev pytest -q <4 node ids in tests/unit/test_diff_coverage.py>
4 failed

diff-cover wasn't installed, and there was no job, rule or patch target yet. Green, at the second commit, the same 4 pass.

Review fixes, red at 72485ef66 (tests only), with the 8 node IDs in tests/unit/test_diff_coverage.py, 4 failed:

• test_the_job_measures_only_the_lines_the_pull_request_changes: 90%, exit 0, with master_change.py counted (should be 50%, exit 1)
• test_coverage_reports_every_module_no_test_imports: the 15 modules in libs/ and routes/ unreported
• test_type_only_imports_and_pragmas_are_not_counted_as_missed: the TYPE_CHECKING import counted as missed
• test_pull_requests_check_the_coverage_of_their_changed_lines: the step still read BASE_SHA

test_a_line_counts_as_covered_if_any_report_covers_it already passed. Green at 1eda3287f: all 8 pass.

Second review round, red at d91472a2e (tests and fixtures only), with the 9 node IDs, 2 failed:

• test_reports_match_the_changed_files_in_another_checkout: "No lines with coverage information in this diff.", exit 0 (should be 50%, exit 1)
• test_type_only_imports_and_pragmas_are_not_counted_as_missed: the #pragma:no cover and # PRAGMA NO COVER lines counted as missed

The proof at 799 of 1,000 lines, and the check that pr.yml runs only on pull_request (which HEAD^1 relies on), passed already. Green at abbfd7c46: all 9 pass.

Third review round, red at 729351590 (tests only): test_only_lines_that_can_run_are_counted failed with missing lines [23] (the Protocol stub) instead of [27] (the untested function whose comment mentions if TYPE_CHECKING:). Green at eedee597a: all 9 pass.

Also run locally:

• the 17 tests of tests/unit/test_codecov_coverage.py from Measure the unit tests' coverage on master and flag every Codecov upload #4196, which read the changed codecov.yml and pr.yml: pass;
• python scripts/run_quality_guards.py: every guard passes;
• python -m scripts.rules guard --base-ref origin/quality/codecov-flags: passes, noting the new rule;
• uv lock --check with uv 0.13.4, ruff check ., ruff format --check ., Codecov's validator, and the changelog fragment check;
• after the two __init__.py files: mypy (190 files, no issues), lint-imports (5 contracts kept), and importing both packages.

Per this workstream's rules, only the new and touched tests were run locally. The rest, including tests/unit/test_ci_complete_gate.py, run in CI once this PR is the lowest in its stack. Until then the full workflow, and so the diff-coverage job itself, hasn't run in CI. The tests above run the job's exact command.

Codecov after this PR

• The integration flag's figure will fall: the 14 libs/ and routes/ modules no integration test imports are now reported, with no hits.
• The unit flag's figure and the combined figure will rise, since if TYPE_CHECKING: imports and stub bodies no longer count as missed. codecov.yml sets no status per flag, so the project status uses the combined figure.
• Unverified until this PR's CI runs: that Codecov resolves the relative <source>policyengine_api</source> to the repository's files. Each name is the repository path once joined with the source, so it should.

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.11%. Comparing base (4c64934) to head (8ab6df6).
⚠️ Report is 14 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4200      +/-   ##
==========================================
+ Coverage   88.63%   92.11%   +3.48%     
==========================================
  Files         304      297       -7     
  Lines       18442    16673    -1769     
  Branches     3084     2719     -365     
==========================================
- Hits        16346    15359     -987     
+ Misses       1357      798     -559     
+ Partials      739      516     -223     

Flag	Coverage Δ
integration	52.27% <ø> (?)
unit	90.25% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.