[anth-volk]

Fixes #4048

Summary

one compact parameter search file per catalog version, served at /v2/parameters/search-file, for the parameter search of the app's newer workspace (v3, app/src/libs/parameterSearch.ts). It is stacked on #4046, and v3 does not use it until it moves to the v2 routes.

• The file (format 1). One JSON document per catalog version: {format, country_id, policyengine_version, parameters, other_parameters, nodes}.
  • parameters: every parameter a reform can change, by Serve the v2 parameter tree one level at a time, with nested bracket schedules and the reform view #4046's shared rule (parameter_tree.is_reform_changeable). Each entry has path, label, unit, description and editable, and, only where it applies, editable_changes.
  • editable says whether a draft can edit the parameter's current-law value: a number, a yes/no value or an infinite threshold, as v3's draft editor allows. Where that changes over time (for example a bracket whose value becomes null from a date), editable describes the earliest value and editable_changes lists {start_date, editable} for each date from which it changes. The file therefore does not depend on the day it was built.
  • other_parameters: the path and label of every other parameter under gov. v3's search reads these for its concept clusters, acronym aliases and member labels.
  • nodes: the path and label of gov and of every node under it that has a stored label, for breadcrumbs, schedule names and concept clusters.
  • Labels are the stored labels, or null. v3's search treats a label that only restates its path's last key as a placeholder, so its reader applies v1's fallback (the key with underscores as spaces, "bracket N" for a bracket).
• Storage. The new parameter_search_files table holds one row per catalog version and file format (unique on both). Each row holds the gzip-compressed JSON (level 9, no timestamp, so the bytes are reproducible), the SHA-256 of the uncompressed JSON, and both sizes. Rows are deleted with their version.
• Revision c614b6bcccfc is chained on 3869eb814a90 (Serve the v2 parameter tree one level at a time, with nested bracket schedules and the reform view #4046). It is autogenerated with no corrections, creates only the table and writes no rows.
• Writing at publication.
  • Catalog publication writes, in its transaction and under its lock, the file of each version it publishes: each new version, and the installed version when an earlier release stored it before search files existed.
  • The file is built from the installed packages' catalog records, never from rows an earlier release stored. It is a row-only, idempotent insert of package-derived rows, which the Alembic skill's deployment-seeder passage allows, as Serve the v2 parameter tree one level at a time, with nested bracket schedules and the reform view #4046's trees are.
  • Writing only inserts. A stored file of the same format must equal the one built from the installed packages, and publication fails if it differs or if a version it publishes is left without a file. A later file format adds a row instead of changing the stored one.
  • Versions stored before catalog format 2 (catalog_format null) get no file. Their parameter rows store no economy or household flags, so a file built from the installed catalog's flags would contradict them. They are exempt from the check above. The stored version's ID and format come from Serve the v2 parameter tree one level at a time, with nested bracket schedules and the reform view #4046's shared publication_reconciliation.stored_version, so both writers apply this rule the same way.
  • The evidence adds parameter_search_files_written (model, version, uncompressed and compressed bytes).
• The route. GET /v2/parameters/search-file?country_id=…&policyengine_version=… (Serve v2 metadata in parts loaded on demand, cacheable per catalog version #4042's strict query model; the version is optional).
  • The body is the file itself, not the status/result envelope.
  • A client that accepts gzip receives the stored bytes unchanged with Content-Encoding: gzip. Any other client receives the uncompressed JSON.
  • Each representation has its own strong ETag, made from the SHA-256 of the uncompressed JSON: "<sha256>", or "<sha256>-gzip" for the gzip one. A 304 needs the tag of the representation being requested, so a client that switches encoding gets a 200.
  • The route runs under Serve v2 metadata in parts loaded on demand, cacheable per catalog version #4042's CacheableMetadataRoute, so Cache-Control (a day for a named version, five minutes for the running one), Vary: Origin, Accept-Encoding, the CORS exposure of ETag and the 304 are those of every metadata route.
  • A request with If-None-Match first reads only the stored hash. When it names the current tag, the route answers 304 without reading the file.
  • A version without a file of the current format returns 503 with a message that names it and says which versions have files. The 503 is not cached. An unknown version returns 404.

Route contract changes

• New route: /v2/parameters/search-file, with its OpenAPI responses (200 with ETag, Cache-Control, Vary and Content-Encoding headers; 304; 404, 405, 422, 500 and 503). The contract registry gains one request (93 in all).

• Every route: gzip follows Accept-Encoding's qualities. Starlette's GZipMiddleware compressed whenever the text "gzip" appeared anywhere in Accept-Encoding. The new content_encoding.NegotiatedGZipMiddleware replaces it application-wide, with the same 1,000-byte threshold and level 4, and decides with accepts_gzip: an explicit gzip or x-gzip entry decides; otherwise * does; a quality of 0 refuses. On every route:

  • a client that refuses gzip (gzip;q=0, br, gzip;q=0.0) now receives an uncompressed response, where it used to receive gzip;
  • Accept-Encoding: *, br, * and differently cased values such as GZIP now receive gzip, where they used to receive an uncompressed response;
  • a value that only contains the text, such as gzipped, no longer receives gzip.

  The search-file route uses the same rule, so its chosen encoding is final and the middleware never re-encodes it. The middleware subclasses Starlette's GZipResponder and IdentityResponder, which Starlette does not document; the locked Starlette is 1.0.0, and a unit test fails with a clear message if either class or its constructor signature changes.

Measured

Locally, with the .py branch containing PolicyEngine/policyengine.py#701 (6.10.0) installed:

Measure	US	UK
parameters (a reform can change them)	92,707	1,540
of which with editable_changes	4	25
other_parameters	7,569	11
nodes	1,960	161
Compressed (stored, and sent as gzip)	868,406 bytes (0.87 MB)	50,736 bytes
Uncompressed	19,941,379 bytes (19.9 MB)	391,059 bytes
Building and encoding the file from the installed catalog	2.6 s, 113.9 MiB peak traced memory	0.04 s

• Route (the US file in local Postgres, FastAPI test client, 15 requests each): gzip 200 in 24.2 ms median (the first request 118.9 ms); 304 revalidation in 9.7 ms median; uncompressed 200 in 25.2 ms median.

• Publication. The production-scale publication test passes with this PR: 52.6 s, and 115.1 MiB (120,693,440 bytes) of additional publisher memory against its 256 MiB limit. Its evidence lists both files with the sizes above.

• Against the plan's estimate. The plan estimated 0.72 MB compressed for the US (92,706 parameters and 29,310 node labels, measured from v1's metadata on 2026-10-21). The file is 0.87 MB compressed and 19.9 MB uncompressed (the plan's measurement: 12 MB). Without other_parameters it is 0.77 MB compressed (774,574 bytes) and 19.0 MB uncompressed.

• other_parameters is kept (decided). v3's search reads labels of parameters a reform cannot change, for its concept clusters, acronym aliases and member labels, so leaving them out could change search results. Keeping them costs 93,832 bytes compressed (0.09 MB).

Staging times need a deploy.

v3's search on this file

v3's own search code (parameterSearch.ts, unchanged) was run over its evaluation cases on three inputs, each a v1-shaped snapshot:

• v1's /{country}/metadata as today (the reference);
• a snapshot rebuilt from this file with every label replaced by v1's label for the same path, so only the file's content differs ("v1 labels");
• a snapshot rebuilt from the file as served, with v1's fallback applied where a label is null ("catalog labels").

Cases	Labels	Found in the top 20: today → file	In the top 5: today → file	Rank changes
US tracker (75)	v1	31 → 31	29 → 29	four cases move one place: ga-hb1001 3 → 2, ga-sb476 4 → 3, ok-sb2156 4 → 3, ut-hb210 2 → 3
US tracker (75)	catalog	31 → 32	29 → 29	ga-sb476, ok-sb2156 and ut-hb210 as with v1 labels; also mo-sb458 5 → 1, ms-sb2869 not found → 11, ny-s9077 16 → 18
US programs (140)	v1	128 → 128	127 → 127	none
US programs (140)	catalog	128 → 128	127 → 127	multnomah_pfa:name 3 → 5
UK queries (48)	v1	48 → 48	48 → 48	none
UK queries (48)	catalog	48 → 48	48 → 48	"income tax" 2 → 3, "scottish higher rate" 1 → 2, "scottish intermediate rate" 1 → 2

No case found today is lost, and the evaluation case files are unchanged.

Labels: .py's labels are kept (decided)

The file carries the catalog's stored labels, which are .py's. .py generates labels for parameters without a stored one (utils/parameter_labels.py): bracket parameters become " (tier N amount)" and breakdown parameters " ()". v1's /{country}/metadata does not have these generated labels: for those parameters it uses the last part of the name with underscores as spaces, and "bracket N" for brackets. The rank changes in the "catalog labels" rows above come from these labels.

This was decided together with #4046's labels: generated labels are shortened only in #4046's parameter tree (/v2/parameters/children). The search file, like the other catalog reads, keeps .py's labels. Having .py also publish the raw stored label was not chosen, so PolicyEngine/policyengine.py#701 and #4023 are unchanged.

Search files for other versions stored earlier: left out

• Publication writes a file only for the versions it publishes, from the installed packages. Writing files for other stored versions would derive rows from data that earlier releases stored, which the Alembic skill's "V2 application-data migrations" section reserves for Alembic and a human migration decision.
• So the installed version gets its file at the first deploy that includes this PR, if it is stored with catalog format 2. Every other stored version returns the 503.
• This follows the decision on Serve the v2 parameter tree one level at a time, with nested bracket schedules and the reform view #4046's parameter trees for other stored versions: v3 asks only for the running version's file, so no backfill is written. A backfill would need its own approved data migration.

Migration notes

• Route group and runtime path. The metadata route group; the route is served by FastAPI and reads Supabase (no change). Nothing moves off Flask.
• Deploy order.
  • The seed job runs alembic upgrade head, then catalog publication, which writes the files, before the new release takes traffic.
  • The previous release does not read the new table.
  • The revision's downgrade drops the table and keeps every catalog row.
• Privileges. The runtime role may read the table. The catalog writer may read and insert rows. A test publishes as the catalog writer role with only its declared privileges, and another reads the file and its hash as the runtime role.
• Environment variables. None are read or added, and no workflow changed.
• Memory. The production-scale publication test, which allows 256 MiB of additional publisher memory, measured 115.1 MiB with this PR.
• Suppressions. One # noqa: E501 is added, on the import of database_connectors.reads_parameter_search_file in the metadata services module: the dotted module path alone is longer than the line limit and cannot be wrapped. It matches the two imports beside it.
• Rule scope and statement change. rules/registry.toml adds policyengine_api/content_encoding.py to the scope, statement and checked paths of api.quality.mypy-v2, and pyproject.toml adds it to mypy's file list, so the new middleware is type-checked with the v2 code. docs/generated/rules.md is regenerated. A review from PolicyEngine/core-developers is requested.

Follow-up (not changed here)

The other metadata routes give their gzip and uncompressed responses one strong ETag, computed before compression. Strong ETags should differ between encodings. This PR only corrects the shared header description ("a hash of the uncompressed response body, so a gzip-encoded response carries the same tag"). Changing those routes is left for a follow-up.

Waits for

• CI on this branch, which runs in full now that Store the metadata v3 reads in the v2 catalog from PolicyEngine.py's public records (catalog format 2) #4023, Serve v2 metadata in parts loaded on demand, cacheable per catalog version #4042 and Serve the v2 parameter tree one level at a time, with nested bracket schedules and the reform view #4046 below it are merged (stack #4155 on master). The branch includes a merge of master (below).
• Review.

The decisions on the labels, on other_parameters and on search files for other stored versions are made (above).

Testing

The evidence below is local. For each commit that changes tests, its changed test modules were run against its parent's code (failed first) and then at the commit (passed). Postgres modules ran on a database recreated at the checked-out code's head, with the managed roles' privileges applied.

Commit	Failed first	Then passed
cb45424f2 Build each catalog version's parameter search file	ModuleNotFoundError for parameter_search_file	11 builder tests
cfd535864 Store each catalog version's parameter search file in its own table	the revision chain, no child revision of 3869eb814a90, the import side-effect test's module list, KeyError: 'parameter_search_files'; lifecycle: UndefinedTable	77 model, revision and import tests; 11 lifecycle tests; alembic check clean
181022df3 Write the parameter search file of each catalog version being published	evidence field missing; no file rows (NoResultFound); a differing stored file did not raise	9 unit and 33 Postgres publication tests
a66c0f1b7 Serve a catalog version's parameter search file at /v2/parameters/search-file	ImportError for the content model and the unavailable error; Postgres: no ETag, 422 instead of 503, no service read	193 unit and 25 Postgres route tests
3d0e74bdd Include the labelled gov node in the parameter search file	assert [] == [{'path': 'gov', 'label': 'Government'}]	22 builder tests
fed8282eb Type the parameter search file and name its editability consistently	ImportError for EditabilityChange	133 tests
2a195cae8 Describe when parameter search file rows are written in the privilege manifest	comment only	the focused set below
7d43be8ea Gzip a response only for a client that accepts gzip	6 tests: gzip;q=0, br, gzip;q=0.0 and gzipped were compressed; br, * and GZIP were not; a large search file sent with gzip;q=0 came back gzip	164 application and route tests
cb3189b44 Name the catalog version in the search file route's 503	the message was "The initialized v2 catalog is unavailable."; the OpenAPI 503 named no file case	165 unit and 25 Postgres route tests
a57d5ab73 Test the search file route's CORS exposure and 404, and its publication roles	tests only; the writer-role test fails with InsufficientPrivilege: permission denied for table parameter_search_files when the grant is revoked and the declaration narrowed	115 unit and 61 Postgres tests
c962328ed Describe the gzip negotiation fix and the search file's 503 in the changelog	documents only	—
8e1afa554 Write no search file for an installed version stored before catalog format 2	a file was written for a null-format installed version	36 Postgres publication tests
f57a986b7 Answer a search file revalidation from its stored hash alone	a matching revalidation read the whole file; no hash read in the service	170 unit and 26 Postgres route tests
6c1b09df0 Declare editable_changes as an optional list that is never null	the shape test, now comparing JSON types, failed on editable_changes	116 route tests
4a54539a2 Pin gzip negotiation precedence, streamed bodies and Starlette's responders	tests only; with the wildcard briefly made to win over an explicit gzip entry, both precedence cases fail	40 tests
c6def889f Say plainly which catalog versions have a search file	the service and Postgres tests failed on the old 503 message	168 unit and 26 Postgres tests
301e361bb Type-check the gzip negotiation module with the v2 code	configuration and documents only	mypy (175 files); rules validate, render and guard
8c54125f4 Look up the stored catalog version through the shared helper	refactor, no behavior change	the focused set below
f364c6459 Split the v2 catalog tables' lifecycle tests under the file-length limit	tests moved only	see "At the head"

At every commit, ruff passes and the focused unit set passes (the search file, metadata route, metadata service, publication, application and gzip modules: 203 to 268 tests). From the second commit on, the Postgres publication and metadata route tests pass at every commit (50 to 62 tests).

On 8c54125f4, before the lock moved to .py 6.12.0:

• make test set (without coverage), under the locked .py 6.10.0 from the lock file: 3,737 passed.
• Postgres tests on a schema recreated at head, grouped as the v2 workflows run them:
  • v2 migration lifecycle and revision tests: 48 passed;
  • after privilege apply and verify: publication and routes 62 passed; user identity and saved-policy view 20 passed; comparison runs and privilege reconciliation 9 passed.
• alembic current --check-heads: c614b6bcccfc (head); alembic check: no new operations.
• ruff, format, mypy (175 files), quality guards and rules validate and guard: clean.
• Rules render and contract export: no change.
• No commit adds a .coverage file or names a private repository.

At the head, under the locked .py 6.12.0, checked before GitHub's rebases onto master (as b1b7c12d8; git range-diff shows every commit unchanged since):

• ruff, mypy (no issues in 190 source files), quality guards: clean.
• Contract export: no change.
• The 74 tests this branch adds or changes (51 unit): 87 unit cases passed.
• Its changed Postgres tests on a schema at head c614b6bcccfc: 23 passed.

Merge of master

• 2f86cf1f1 merges master at 8c3fb8495, which holds Serve the v2 parameter tree one level at a time, with nested bracket schedules and the reform view #4046 with its own merge of master: the tree revision 3869eb814a90 now follows e86f19a6c4da (the UK area impact tables' map coordinate removal), and the v2 lifecycle tests are split as this branch split them. The search file revision c614b6bcccfc stays on 3869eb814a90 and is the single v2 head.
  • test_alembic_v2_lifecycle.py is master's: this branch's only change to it was the search file test, which its split had already moved.
  • test_alembic_v2_catalog_tables.py is master's module with this branch's search file test, so the tree test downgrades to e86f19a6c4da.
  • The revision chain test lists 3869eb814a90, then e86f19a6c4da, between the head and the catalog format 2 revision.
• Evidence, locally on the head: the revision chain and tree revision unit tests and the tree and search file lifecycle tests passed; alembic heads shows c614b6bcccfc alone; alembic check finds no new operations; ruff, mypy (190 files), the quality guards, rules validate and guard, uv lock --check and the contract export pass. One review round ran on the merge (below).

Review

• Round 1 found two minor bugs.

  • A client that refused gzip (gzip;q=0) still received gzip under the uncompressed form's ETag, because Starlette's middleware compressed the body. Fixed in 7d43be8ea, which changes negotiation on every route (above).
  • The file left out the labelled gov node ("Government"), which v3's concept clusters read. Fixed in 3d0e74bdd.

  Its quality findings led to the move onto Serve v2 metadata in parts loaded on demand, cacheable per catalog version #4042's caching route class (in the route commit, during the rebase onto Serve v2 metadata in parts loaded on demand, cacheable per catalog version #4042 and Serve the v2 parameter tree one level at a time, with nested bracket schedules and the reform view #4046), fed8282eb, 2a195cae8, cb3189b44 and a57d5ab73. 8e1afa554 then applied Serve the v2 parameter tree one level at a time, with nested bracket schedules and the reform view #4046's rule for versions stored before catalog format 2.

• Round 2 found no substantiated bugs. Its final-pass findings are fixed in:

  • f57a986b7: the 304 reads only the hash;
  • 6c1b09df0: editable_changes is optional and never null, and the shape test compares types;
  • 4a54539a2: tests for precedence, streamed bodies, Starlette's responders and an editability change in the middle of a value list;
  • c6def889f: the 503 message, the changelog fragments and the shared ETag description;
  • 301e361bb: the mypy file list;
  • 8c54125f4: Serve the v2 parameter tree one level at a time, with nested bracket schedules and the reform view #4046's shared stored-version helper.

• Merge of master: one review round on 2f86cf1f1 found no substantiated bugs and no material quality findings. It noted that the search file revision's creation date is earlier than its regenerated parent's; its operations touch only parameter_search_files, and the lifecycle tests confirm the head schema matches the models, so it is not regenerated.

🤖 Generated with Claude Code

[vahid-ahmadi]

Review at 0e6bd7f864.

Verdict: correct and well tested. Two small points on the new gzip negotiation, and one operational consequence to write down. Stacked on #4046, so I reviewed only v2plan/1f-117-parameter-children...v2plan/1f-118-search-file. No must items.

1. should policyengine_api/data/v2/catalog/publication_parameter_search_file.py:109 together with the 503 for "no file of the current format": any change to how the file is built (an editable fix, a new field) has only two outcomes. Without a format bump, publication fails on the installed version and the deploy stops. With a bump, every older named version answers 503 until something backfills it, because only the installed version gets a new file. Both are safe, but the second is a visible regression for the app's pinned versions. Worth stating in docs/migration/stage-9-v2-metadata.md, next to the format paragraph, so the next person to touch the builder knows which one they are choosing.
2. nit policyengine_api/content_encoding.py:45: the first explicit gzip entry decides, so gzip;q=0, x-gzip returns False even though x-gzip is accepted (checked locally). This is rare in practice; taking the highest quality across both names would match RFC 9110.
3. nit policyengine_api/fastapi_routes/v2/metadata/search_file_response.py:74: a client that does not accept gzip makes the route decompress the full 19.9 MB US file in memory on every 200. Browsers always accept gzip, so this only affects scripts, but with 6 concurrent requests an instance holds about 120 MB for it. A cap or a streamed decompression would bound it.

Checked and fine: the gzip response sets Content-Encoding, so the middleware passes it through untouched; the ETags differ per representation and Vary includes Accept-Encoding; If-None-Match reads only the hash; hashing the uncompressed JSON means a zlib change cannot fail publication. The application-wide middleware change is a correct tightening for v1 too: gzip;q=0 now gets identity.

What I ran: fresh clone, uv sync --frozen --extra dev, then pytest on the changed unit and contract test files: 355 passed, plus a quick check of accepts_gzip on six header values. CI on this head was cancelled (CI complete reports the cancelled jobs, not test failures). I did not run the integration or production-scale publication tests, or v3's search evaluation.

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 93.13433% with 23 lines in your changes missing coverage. Please review.
✅ Project coverage is 88.30%. Comparing base (8c3fb84) to head (2f86cf1).
⚠️ Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
...ta/v2/catalog/publication_parameter_search_file.py	82.60%	6 Missing and 2 partials ⚠️
...ngine_api/data/v2/catalog/parameter_search_file.py	96.49%	3 Missing and 1 partial ⚠️
...api/fastapi_routes/v2/metadata/parameter_routes.py	85.00%	2 Missing and 1 partial ⚠️
policyengine_api/content_encoding.py	94.59%	1 Missing and 1 partial ⚠️
...icyengine_api/fastapi_routes/metadata_protocols.py	50.00%	2 Missing ⚠️
...fastapi_routes/v2/metadata/search_file_response.py	89.47%	1 Missing and 1 partial ⚠️
...database_connectors/reads_parameter_search_file.py	83.33%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #4049       +/-   ##
===========================================
+ Coverage   53.29%   88.30%   +35.01%     
===========================================
  Files         276      293       +17     
  Lines       15936    17457     +1521     
  Branches     2579     2865      +286     
===========================================
+ Hits         8493    15416     +6923     
+ Misses       6730     1331     -5399     
+ Partials      713      710        -3     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.