Repository navigation
Epic: publish the US default from the graph-native build #956
Description
Tracks the path to publishing a certified US default built by the graph-native line, and the acceleration work along it. Opened 18 September 2026 from the data weekly's ask for an updated graph timeline.
Definition of done
tools/publish_release.sh runs on a certified US release built by the graph-native line; #893 is merged into main only after a from-scratch build, preflight and certification have run on its own tree (the rule in docs/us-release-build-rule.md, #947); main carries the same build as a standing check.
Decided 27 September 2026: the certified build reads no artifact built by policyengine-us-data code. Every input is a raw public or licensed source file pinned by digest, or an artifact microcosm builds from those files. This replaces #983. On main today the legacy base stage (tools/build_us_puf_support_base.py) still reads five such artifacts:
puf_2024.h5fromrelease://policyengine/irs-soi-puf/1.8.0(--puf-h5);- the 2022 ACS rent donor
acs_2022.h5(--acs-h5); census_cps_2022.h5,census_cps_2023.h5andcensus_cps_2024.h5, mirrored unchanged topolicyengine/microcosm-us-sources(--asec-h5).
The graph-native branch already reads the raw 2024 ACS PUMS and builds its PUF donor from the raw 2015 file with its own growth recipe (us_runtime/puf_target2024_growth.py). Whether any native node still reads the processed ASEC files is not yet checked. The criterion is met when the build receipt's source roster names none of the five files above and no other policyengine-us-data output.
Where we are (18 September, evening)
- The stacked line Consolidate the US survey graph build and enrichment pipeline #893 ← Verify native sources once per run, not once per access and per node #935 ← Lift the whole-roster receipt ceiling and vectorise the frame seal #945 ← Lift the seven row-count ceilings a full-source native build meets, under one rule #949 / Compare replayed populations by content seal instead of by retained object #950 is merged level with main; every PR in it is mergeable. Let a seal-only population observer skip the executor's snapshot detachment #951 (executor observer flag) and Record on main that main builds the certified US default from raw sources #947 (the rule and its four blockers) are merged.
- Shared inputs: Pin the processed CPS ASEC inputs to their public mirror and verify them in the base build #952 (ASEC pins, one gate round from merge), Re-pin the US Chronicle feed to a labelled export rebuilt from targeted package runs #955 (labelled Chronicle feed re-pin, CI running), the SPM independence role as a build stage (six commits on
us-spm-role-stage, lane in progress), fresh lineage accepted, build-bundle emits source_label concept alignments without authority; build-consumer-artifact refuses them under consumer_fact.v3 chronicle#277 filed for the consumer-artifact refusal. - Measured: the 19-node graph completes at 1/1000 (2,028 CPU-s, 84% of it US-runtime source authentication outside the graph; a required replay with every node a store hit still costs 1,664 CPU-s). Nothing above 1/1000 has completed. The executor runs nodes sequentially. Modal is not wired for US runs.
Steps
Type: T = agent-bound, C = compute-bound, H = a decision.
- 1. Byte transports lifted (lane on
native-byte-transports, ACS budgets first) and the 1/15 run completes on its head — T then C, 1–2 days - 2. 1/10 run — C; skipped unless the 1/15 run meets a ceiling the census missed
- 3. Stack merged downward through gates: byte transports → Lift the seven row-count ceilings a full-source native build meets, under one rule #949 → Lift the whole-roster receipt ceiling and vectorise the frame seal #945; Compare replayed populations by content seal instead of by retained object #950 → Lift the whole-roster receipt ceiling and vectorise the frame seal #945; Lift the whole-roster receipt ceiling and vectorise the frame seal #945 → Verify native sources once per run, not once per access and per node #935; Verify native sources once per run, not once per access and per node #935 → Consolidate the US survey graph build and enrichment pipeline #893 — T + H, 1–2 days
- 4. Shared inputs on main:
Pin the processed CPS ASEC inputs to their public mirror and verify them in the base build #952(merged 22 Sep), Re-pin the US Chronicle feed to a labelled export rebuilt from targeted package runs #955 (in its merge gate 22 Sep; Accept a restated Ledger filter bound only where it agrees with the compiled constraint #969 stacked on it), the SPM stage PR (Materialize the Census SPM independence role in US builds #959) — T + H, 2–4 days - 5. Consolidate the US survey graph build and enrichment pipeline #893 re-levelled on main so the graph build reads the labelled feed, the pinned ASEC inputs and the role stage — T, hours
- 6. Full-source native run on Consolidate the US survey graph build and enrichment pipeline #893's tree — C; 15–76 CPU-h on the sequential executor (1–3 days wall); the ledger gives a first attempt 25% and three attempts 70%, so plan for two: 3–6 days with fixes. Critical path.
- 7. Preflight and certification on that output (release gates, reform-coverage smoke, SPM composition, target parity) — C + H, about a day plus decision 1 below
- 8. Consolidate the US survey graph build and enrichment pipeline #893 gate with the build receipt attached; merge — T + H, 1 day
- 9. Standing from-scratch check on main — T to write, C to run
- 10. Publish — H
Serial sum about 3–5 weeks. Current estimate (my ledger, 18 September): 10% by 1 October, 35% by 15 October, 60% by 1 November; median about 18 October. A Codex second opinion is planned for 19 September.
Acceleration, by leverage
- A. Publish from main's tools first. When Pin the processed CPS ASEC inputs to their public mirror and verify them in the base build #952, Re-pin the US Chronicle feed to a labelled export rebuilt from targeted package runs #955 and the SPM stage land, run the base stage (46 min measured) and the release tool on the labelled feed and publish that certified release. The graph's launch criterion then becomes reproducing the certified default's receipts. Needs the operator's word for the build.
- B. Parallel executor. Independent DAG branches execute concurrently behind a flag, with node keys, receipts and store objects byte-identical to the sequential run. Converts step 6 from days per attempt toward the longest chain. Lane brief written; PR against main.
- C. Cut the 84%. Attribute the outside-graph source-authentication cost precisely and memoize what is content-addressed across nodes and runs. Same lane as B.
- D. Skip the 1/10 run (step 2) unless the 1/15 run says otherwise.
- E. A second machine or Modal for full runs, so fix attempts run concurrently instead of serially on the one 128 GiB box. Modal is not wired for US runs; 1–2 days.
- F. Lane capacity: every reactivated Claude account is a lane; Codex returning gives Astra for the hard reviews.
Where we are (22 September, morning)
- Pin the processed CPS ASEC inputs to their public mirror and verify them in the base build #952 and us: name the SPM composition defect in seconds, and refuse it by name in the release #948 merged. Re-pin the US Chronicle feed to a labelled export rebuilt from targeted package runs #955 peer-approved (round 2, no findings) after its release build learned to hold the feed to the committed pin; re-levelled on main again and merging on green CI. Accept a restated Ledger filter bound only where it agrees with the compiled constraint #969 (restated-filter guard the labelled feed needs on main's compiler) stacked on it, to re-level after. Recover ACS usual hours and validate local release inputs #923 peer-approved and merging on green CI; Gate the ACS local-area finalize on the usual-weekly-hours surface (#765) #941 in its review gate, to re-level on Recover ACS usual hours and validate local release inputs #923.
- A full-scale local ACS local-area build ran on 22 September on a tree carrying Recover ACS usual hours and validate local release inputs #923 + Gate the ACS local-area finalize on the usual-weekly-hours surface (#765) #941 (the acs-local: weekly_hours_worked_before_lsr defaults to 40, silently nulling SNAP ABAWD analysis #765 fix), the Medicaid hierarchy fix from Re-pin the US Chronicle feed to a labelled export rebuilt from targeted package runs #955, a donor qualified by Qualify a pinned Build P lineage donor with the reported-receipt inputs the ACS staging gate requires #972 and the package guard from Refuse to package a capped ACS local-area staging run #973: every finalize gate green at full geography; on the ACS spine usual weekly hours went from one distinct value (all 40, the published Build O) to 100 distinct values. Receipt: Record the full-scale local ACS hours rebuild of 2026-09-22 (#765 build evidence) #974 (
experiments/us-acs-local-hours-rebuild-20260922/); the donor qualification tool is Qualify a pinned Build P lineage donor with the reported-receipt inputs the ACS staging gate requires #972 and the capped-smoke package guard is Refuse to package a capped ACS local-area staging run #973; the group-quarters housing take-up flag the build exposed is ACS group-quarters records draw a housing take-up flag in the local-area transfer #975. Not published; the target contract used SOI totals mode (1,247 targets), which needs a ruling before anything is called a release candidate. This is acceleration item A's "main's tools can build" half; the graph's launch criterion (reproduce the certified default's receipts) is unchanged. - SPM consumer scope, measured (counts only): the ACS spine carries 5,246 group-quarters one-person units of minors no role can classify (Census excludes group quarters from the ACS SPM universe); the published ACS-local file has the same units. Decision 3 below.
- The 1/15 native run completed on 19 September (19 nodes, 8,015 wall-s, 46.7 GB peak).
Decisions
-
The certification claim at step 7. A "certified release" is available on the
--base-h5arm now. Promotion as the default under US base v2: one CPS+ACS+PUF-detail pool; datasets labeled by exact record count (dense = full pool; exact-k L0 selection) #578's improvement gate needs the--exact-karm, which needs a Chronicle consumer artifact (build-bundle emits source_label concept alignments without authority; build-consumer-artifact refuses them under consumer_fact.v3 chronicle#277). Recommendation: publish as a certified release; let US base v2: one CPS+ACS+PUF-detail pool; datasets labeled by exact record count (dense = full pool; exact-k L0 selection) #578 promotion follow the Chronicle fix. -
The SPM measurement universe. Ruled 21 September: the engine-side universe rule is the fix (Separate SPM measurement scope from ordinary income policyengine-us#9462, re-levelled on main 22 September; only the "published default declares no scope" failures remain). Microcosm side:
docs/us-spm-measurement-universe-producer.mddesign (ACSTYPEHUGQ1 → INCLUDED, 2/3 → OUTSIDE; every ASEC record INCLUDED). Engine sub-decision ruled 22 September (Max): strictUNRESOLVEDdefault, with the package'sDEFAULT_DATASETand its sha256 moving in the same release, so #9462 lands together with the first default that declares scope; no transition rule. Microcosm-side rulings the same day on Declare the US SPM measurement universe from the source record type #977: the producer module stays out ofPRODUCER_SOURCE_FILES; the missing-SPM_IDrefusal stays strict;puf_tax_detailstays refusing until its arm is established; every ASEC record isINCLUDEDvia a named, cited constant with its own refusal. Read the SPM poverty indicator as nullable in _person_rate #976 (nullable_person_rate) merged 22 September after M4a showed all 104 state SPM rates bit-for-bit unchanged. -
Merge order for the stack (step 3): downward through gates as listed, or wait and merge Consolidate the US survey graph build and enrichment pipeline #893 with everything at once after the build. Recommendation: downward now, so each piece carries its own review and the final gate reviews only the build receipt.
Not in scope
The UK graph work (María's calibration-on-graph PR, paused on #893) and the geography reordering (deferred until after the Autumn Budget, one-off test first) are tracked separately.
🤖 Generated with Claude Code
Activity
MaxGhenis commentedon Sep 19, 2026
September 19 continuation update, 16:00 UTC:
The native 1/1000 cold and required-cache-replay pair passed at source
9af56aa8a7a210e18674d484b11d36f417de6759. All 19 financial and 9 prefix replay nodes were cache hits. Manifest keys, canonical content, entity counts and all 5,869 store files matched: zero added, changed or removed files. Output: 3,168 households and 6,928 people. Cold took 1,999.41 CPU seconds / 2,081.53 wall seconds; replay took 1,652.08 CPU seconds / 1,716.61 wall seconds. Whole-process peak RSS was 11,900,780,544 bytes. Aggregate report SHA256:4e7be8adfe472a684cd8ff86a7139cffb42e8a3699d8072d02775fdd9554976d. That job is complete.Statistical CPU attribution puts graph calls at 317.74 seconds cold versus 27.12 seconds on replay; source preparation and other work outside graph calls still take 1,624.74 CPU seconds. Another task had a concurrent population calculation, so wall timings are observational. A narrow ACS parser improvement now reuses exact-string membership indices across chunks while preserving age, orphan and roster checks. Local commit
fee4aac9d7be2643fe02ec6b855d231abe5980a4passed 185 focused tests and independent review. An invented benchmark confirms the repeated representation work is removed; no native speedup is claimed yet.Completion-compatible memory reduction is under final synthetic testing: private directional observation witnesses release intermediate snapshots while keeping live boundary objects, mutation checks, source reauthentication and complete union-state derivation. The default executor/store behavior is unchanged. The 45-, 49- and 51-node completion cases are running in separate small test processes. A full 51-node native comparison is being prepared with separate all/compact cold/required processes; it has not run. The older 1/15 experiment remains 211,638 households / 475,728 people, 8,015.20 seconds wall and 43.46 GiB peak RSS: one
resume=autoresult, not required replay or certification. Full-source memory fit remains unproven.ACS SPM structural qualification now passes on actual pilot data. Source
4d567e2e04bb6494ae895638fe5f675bda4bc0f7and exact calculator PR45 source reconstructed 1,183 people from 512 old units into 542 SPM units, preserving all 62 group-quarters units. Both partner-role sensitivities and both tenure policies match the saved membership/regroup goldens. Eight source-uncertain households remain explicit; 21 childcare ledger rows reconcile. Every primitive table and authenticated input/control file is unchanged. No country engine, network or unapproved population read occurred. Runtime was 9.739 seconds, peak RSS 366,772,224 bytes. Report SHA256:2409e8b410706f9b9c6ff857f06ca7a4051eee11019e1a81046419677784cf1d.The first attempt exposed a golden-comparison bug between pandas nullable Booleans and JSON nulls; its failure remains preserved. Aggregate diagnosis found zero value differences in all 1,183 × 10 membership cells. The comparison-only fix checks missing masks, distinguishes Boolean from numeric values, and preserves exact nonmissing values, including integer precision. All 28 focused harness tests pass. The unchanged helper matrix has 173 passes with reviewed PR45 source; actually absent and installed-1.0.0 assemblers each have 132 passes and 41 strict expected refusals, zero skips/unexpected failures. The four source/harness review findings are closed. Implementation Fable review remains pending; this is a development source proof, not annual-universe, country-consumer or release acceptance. The source and public aggregate proof are now draft #962 at
9024be75da1117050eaf774fc5b2efef9bd35df3; CI is running. That later head adds documentation; the actual passing run stays pinned to4d567e2e. A separate opt-in source adapter is now being built to construct canonical SPM membership before the initial graph CREATE, preserving tax linkage and household weights; it has no actual-data result yet.SPM source-role integration remains draft #959, stacked on #948, head
96761c63526eaca6a17e3bf0cc5109882edebe7f. Actual wrapper/gate proofs at implementation6a6d53b2fb2cad7ac7b84634decc2a41125efa92pass on Build P (166,321 people) and base-q3 (907,382 people): roles match the pinned Census derivation, 28 and 222 unresolved units become zero, and existing inputs/data remain unchanged. Gate JSON serialization is exercised. Fable closed eight prior findings with no new high/medium defect. All nine selected generated-contract files passed across focused runs. Its feature-branch base does not trigger main-only PR CI.The country-model reconciliation now has a clean local source result at
6f6d7d33ca6a7e37508bb94427ec0721d1c02f82: 167 final affected tests, 44 YAML cases and 13 additional baseline YAML checks pass, with both independent reviews approving after a county-typing collision was fixed. It preserves main's separation of ordinary housing benefits from SPM valuation. Exact Microdf compatibility was checked on the same invented fixtures: 1.3.0 passes all 43 required checks, while locked 1.2.1 fails 10, principally missing-aware counts. Both already handle direct missing-aware means. Tied-income ranks change; the wrapper already locks 1.3.0 and documents that tie convention. A minimal isolated country pin/lock candidate and a bounded 512B consumer harness are being prepared. Generic Gini-shift and missing-quantile limitations remain separate and no blanket dependency/no-worse claim is made. The old conflicting #9462 has not been substituted or merged. Actual population/wrapper/universe qualification remains outstanding.Chronicle publisher-authority fix PolicyEngine/chronicle#278 passed both exact-head CI runs and is ready for review, not merged. The valid US artifact contains 39,158 rows and 586 record-set/period pairs. Microcosm consumer draft #961, stacked on #955, is at
112a2d2069bf7a1bc8e83b75019cd9fd84566422. All fact values and 32,867 target identities/hierarchies remain unchanged; 994 fact rows and 166 targets have expected provenance changes. Facts SHA2564d1dba8c1b6274877bf184fa6de5d99b13fc61f34709ccab1487db2b5c64a79f, manifest SHA25638ec5bf1efe5a0bd017ec5279065e2ea7645b37da237197f03ae2fbca28cadac. Its 195-test integration battery, two reporting regressions and all 13 renewed real-artifact comparison checks pass. Fable reporting findings are closed. The stacked consumer has no main-only PR CI run.The #958 owner continues #960 targets plus full-vector tail support; #960 now has green CI, but target-only adoption remains unqualified. The pre-existing ten-year tail calculation finished, freeing memory for the next native study. The #779 owner controls the separate humanitarian-status integration and affected schema/spec pins. No new certified file, incumbent no-worse result, promotion or publication is claimed. Release-critical work remains SPM consumer/dependency qualification, full native graph execution within local resources, calibration and matched national/congressional-district acceptance.
MaxGhenis commentedon Sep 22, 2026
State on 2026-09-22 midday:
MaxGhenis commentedon Sep 23, 2026
State on 2026-09-23, 05:30 EDT:
Merged since the last note: #969 (restated Ledger filters), #980, #985 (the reported-receipt enrichment as a
source_enrichmentoperation), #984 (the ACS local default SOI surface isstate, Build O/P's 4,459-target contract;totalsandfullare opt-in), #986 (restated age-band filter), #977 (SPM measurement-universe producer, not wired into any build), #988 (new-lineage mode for the release-gate preflight).Published overnight, tag-only with
--no-latest, after every gate passed (latest.jsonstill points atpopulace-us-2024-spm-20260915):populace-us-2024-spm-receipts-20260923: the reported-receipt child of the national default (sha25600946972…), certified against the four pinned wheels.populace-us-2024-buildo-acs-local-767312d60-20260923T074941Z(sha256769756c3…), built on that donor with thestatesurface: calibration loss 0.0155, 97.58% of targets within 10%, Kish ESS 13,631, and all eight finalize gates green. It fixes acs-local: weekly_hours_worked_before_lsr defaults to 40, silently nulling SNAP ABAWD analysis #765's constant 40-hour usual hours on the ACS spine. Tracked as the non-default local-area file.Acceleration A (publish from main's tools) is running. The driver makes a local release candidate and never publishes.
47976be6c): the loader reads the pinned member in the archive and checks the archive and member digests.--base-h5, no selection source) started at 05:16. Its runtime at this scale has not been measured before.In review: #959 (SPM role stage; review found no code defect; doc fixes in progress; the merged code will be the code the route A build ran), #987 (a US build stage on Modal; hardening fixes in progress), #990 (the preflight crosswalk fix; peer-approved, merging on green CI), #989 (draft record: 100-record exact-k against PolicyBench's sampled 100 on the same release).
Modal (acceleration E) measured. The acceptance replay of the full-scale materialize was preempted twice (at 54 min and 2 h 4 min) and was stopped at about $3.62 list price without a receipt. Per chunk it ran 3.3 to 7.6 times slower than the local machine, bound to one core. #987 adds an opt-in non-preemptible placement at three times the list price. Uploading the restricted PUF inputs to Modal for the base stage awaits a decision.
SPM measurement scope (decision 3): no release can declare
spm_unit_spm_universe_statusyet. PolicyEngine/policyengine-us#9462 gives the variable a formula (householdsINCLUDED, datasetsUNRESOLVED), and Microcosm's adapter does not export formula-owned variables. The design note's Q-A (an upstream source-declaration marker, with a Microcosm allowlist as a bridge) awaits a ruling, and until then #9462 cannot land with the default.🤖 Generated with Claude Code
Merge main (6442030) and poverty-comparison-only (07b69be) into the n…
Bind terminal-gate evidence into the release manifests (Route A PR-3)
Bind terminal-gate evidence into the release manifests (Route A PR-3)
Score the post-export release gates over household batches (Route A P…
Fence reviewed exclusions at every vintage and restore the M-CHIP CHI…
Bind the staged frame into the target-frame checkpoint identity (Rout…
Batch the target materializer's base simulation (Route A PR-5, F-1)
Arm the batch-invariance guard for pre-sliced callers and every engin…