From 003dbd0062cdb43f6a1eafd21e1f33a761bdfb46 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Mar=C3=ADa=20Juaristi?= <127882282+juaristi22@users.noreply.github.com> Date: Thu, 29 Oct 2026 14:27:10 +0000 Subject: [PATCH 1/2] Make the promoted national line the default UK load microcosm#823 (tracked in #1095). The registry kept the UK 2024-25 national line off the default variant until its first certified cut existed on the Hub, so that resolve("uk") never chased an absent pointer; its comment named the flip a one-line follow-up after promotion (review of #966, Max's #823 ruling). Checked on 2026-10-29: latest-national.json on policyengine/populace-uk-private names microcosm-uk-2024-25-national-20261025T091414Z-a6299504 (updated 2026-10-26), with its release certification, built on policyengine-uk 2.151.0, the version #1095 names for the flip. The entry's variant becomes DEFAULT_VARIANT, so resolve("uk") and load("uk", 2025) follow latest-national.json, and load("uk", 2023) stays on latest.json. This changes the default UK load for every consumer of microcosm.data; no other local repository names the retired national variant. The registry, loader and assembler tests and the release runbook follow; the live pointer test now branches on the line pointer, not the variant, and resolves the promoted cut. The unreleased changelog entry from #966 that described the opposite state goes. It moves out of #1422 on its own, so that it can land now (Vahid's review of #1422). Co-Authored-By: Claude Opus 5.5 --- .../823-national-line-default-load.changed.md | 1 + ...national-variant-until-promoted.changed.md | 1 - ...k-national-release-assembly-runbook-806.md | 12 ++-- .../uk/test_uk_release_assembler.py | 4 +- .../src/microcosm/data/registry.py | 14 ++--- .../tests/engine_free/shared/test_loader.py | 58 +++++++++---------- .../tests/engine_free/shared/test_registry.py | 22 +++---- 7 files changed, 54 insertions(+), 58 deletions(-) create mode 100644 changelog.d/823-national-line-default-load.changed.md delete mode 100644 changelog.d/823-national-variant-until-promoted.changed.md diff --git a/changelog.d/823-national-line-default-load.changed.md b/changelog.d/823-national-line-default-load.changed.md new file mode 100644 index 0000000000..43b06fe7d0 --- /dev/null +++ b/changelog.d/823-national-line-default-load.changed.md @@ -0,0 +1 @@ +The UK 2024-25 national line is the default UK load: `microcosm.data.load("uk")` and `load("uk", 2025)` follow `latest-national.json`, which has named the certified cut microcosm-uk-2024-25-national-20261025T091414Z-a6299504 (built on policyengine-uk 2.151.0) since 26 October 2026, instead of the June 2023 `populace_uk_2023` artifact, which stays at `load("uk", 2023)`. The line's own `national` variant key retires (microcosm#823, Max's ruling). diff --git a/changelog.d/823-national-variant-until-promoted.changed.md b/changelog.d/823-national-variant-until-promoted.changed.md deleted file mode 100644 index c492e22e40..0000000000 --- a/changelog.d/823-national-variant-until-promoted.changed.md +++ /dev/null @@ -1 +0,0 @@ -The UK 2024-25 national line is registered as variant `national` (`microcosm.data.load("uk", 2025, variant="national")`), not as the 2025 default: `resolve("uk")` stays on the June 2023 line until the line's first cut is promoted and a follow-up flips the entry to the default variant, so a default load never chases a pointer that does not exist yet (review of #966; Max's #823 ruling keys the line as a variant). diff --git a/docs/uk-national-release-assembly-runbook-806.md b/docs/uk-national-release-assembly-runbook-806.md index bec7ea1896..38a19e3e6c 100644 --- a/docs/uk-national-release-assembly-runbook-806.md +++ b/docs/uk-national-release-assembly-runbook-806.md @@ -222,13 +222,11 @@ in ยง4. The promotion moves only `latest-national.json`, recording `national` as the line and the reviewed cut tag as its `revision`. It never moves the repository-global `latest.json`, which remains on the June 2023 release. -The 2025 national entry is registered off the default variant until this -first promotion: `microcosm.data.load("uk", 2025, variant="national")` follows -`latest-national.json` as soon as the pointer exists, while -`microcosm.data.resolve("uk")` stays on the June 2023 line. The default moves -in a one-line follow-up PR after promotion (the registry entry's `variant` -becomes the default), with its test; sequencing it this way means a default -load never chases a pointer that does not exist yet. An explicit +The 2025 national entry was registered off the default variant until the +first promotion, so that a default load never chased a pointer that did not +exist yet. The first certified cut was promoted on 2026-10-26, and the entry +is now the default variant (microcosm#823): `microcosm.data.load("uk")` and +`microcosm.data.load("uk", 2025)` follow `latest-national.json`. An explicit `microcosm.data.load("uk", 2023)` continues to follow `latest.json`. With `SLACK_WEBHOOK_POPULACE_UK` configured, successful promotion sends an diff --git a/packages/microcosm-build/tests/engine_free/uk/test_uk_release_assembler.py b/packages/microcosm-build/tests/engine_free/uk/test_uk_release_assembler.py index 5c325d86a6..68f4fc644a 100644 --- a/packages/microcosm-build/tests/engine_free/uk/test_uk_release_assembler.py +++ b/packages/microcosm-build/tests/engine_free/uk/test_uk_release_assembler.py @@ -28,7 +28,7 @@ uk_weight_projection_factor, ) from microcosm.data.contract import validate_release_dir -from microcosm.data.registry import REGISTRY +from microcosm.data.registry import DEFAULT_VARIANT, REGISTRY from microcosm.frame import WeightKind from test_support.paths import paths_for @@ -87,7 +87,7 @@ def test_national_dataset_filename_mirrors_registry() -> None: assert driver._DATASET_KEY == "microcosm_uk_2024_25" assert driver._DATASET_FILENAME == f"{driver._DATASET_KEY}.h5" - assert REGISTRY[("uk", 2025, "national")].filename == driver._DATASET_FILENAME + assert REGISTRY[("uk", 2025, DEFAULT_VARIANT)].filename == driver._DATASET_FILENAME def _frame(weights: list[float], *, weight_kind: WeightKind): diff --git a/packages/microcosm-data/src/microcosm/data/registry.py b/packages/microcosm-data/src/microcosm/data/registry.py index e16e9f45fa..385a062c01 100644 --- a/packages/microcosm-data/src/microcosm/data/registry.py +++ b/packages/microcosm-data/src/microcosm/data/registry.py @@ -114,15 +114,15 @@ def register(spec: DatasetSpec) -> DatasetSpec: register( DatasetSpec( # microcosm#823: the UK 2024-25 national line, followed through its - # own pointer. Registered OFF the default variant until the first - # promoted cut exists on the Hub: resolve("uk") picks the latest - # default-variant year at import time, so a default entry here would - # send every default load to a pointer that does not exist yet. The - # flip to DEFAULT_VARIANT is a one-line follow-up after promotion - # (review of #966; Max's #823 ruling keys the line as a variant). + # own pointer, is the default UK load. It stayed off the default + # variant until its first certified cut was promoted (review of #966; + # Max's #823 ruling), and latest-national.json has named + # microcosm-uk-2024-25-national-20261025T091414Z-a6299504 since + # 2026-10-26. resolve("uk") picks the latest default-variant year, so + # it loads this line; the June 2023 artifact stays at ("uk", 2023). country="uk", year=2025, - variant="national", + variant=DEFAULT_VARIANT, hf_repo="policyengine/populace-uk-private", filename="microcosm_uk_2024_25.h5", engine_module="policyengine_uk.data", diff --git a/packages/microcosm-data/tests/engine_free/shared/test_loader.py b/packages/microcosm-data/tests/engine_free/shared/test_loader.py index 69c5af1096..decabfd3d3 100644 --- a/packages/microcosm-data/tests/engine_free/shared/test_loader.py +++ b/packages/microcosm-data/tests/engine_free/shared/test_loader.py @@ -345,23 +345,22 @@ def __init__(self, *, file_path: str) -> None: return constructed -def test_resolve_uk_default_stays_on_2023_until_the_national_line_is_promoted(): - """The national line is registered off the default variant until its - pointer exists on the Hub, so a default load never chases an absent - pointer (review of #966); the flip is a one-line follow-up.""" - assert latest_year("uk") == 2023 - assert resolve("uk").key == ("uk", 2023, DEFAULT_VARIANT) - assert resolve("uk", 2025, variant="national").pointer_path == ( - "latest-national.json" - ) - with pytest.raises(ValueError, match="published years"): - resolve("uk", 2025) +def test_resolve_uk_default_is_the_promoted_national_line(): + """microcosm#823: once the national line's first certified cut was + promoted, the line became the default UK load; the June 2023 line stays + addressable at its year and its own pointer (Max's #823 ruling).""" + assert latest_year("uk") == 2025 + assert resolve("uk").key == ("uk", 2025, DEFAULT_VARIANT) + assert resolve("uk").pointer_path == "latest-national.json" + assert resolve("uk", 2023).pointer_path == "latest.json" + with pytest.raises(ValueError, match="variant 'national'"): + resolve("uk", 2025, variant="national") def test_resolve_uk_national_variant_follows_the_line_pointer( tmp_path: Path, ) -> None: - spec = resolve("uk", 2025, variant="national") + spec = resolve("uk", 2025) manifest = _uk_release_manifest() hub = _line_pointer_hub( tmp_path, @@ -374,7 +373,7 @@ def test_resolve_uk_national_variant_follows_the_line_pointer( certified = loader._resolve_certified_release(spec, hub_download=hub) manifest_path = f"releases/{UK_NATIONAL_RELEASE_ID}/release_manifest.json" - assert spec.key == ("uk", 2025, "national") + assert spec.key == ("uk", 2025, DEFAULT_VARIANT) assert spec.pointer_path == "latest-national.json" assert certified.release_id == UK_NATIONAL_RELEASE_ID assert certified.artifact_revision == UK_NATIONAL_CUT_TAG @@ -449,9 +448,7 @@ def test_line_pointer_refuses_any_artifact_revision_mismatch(tmp_path: Path) -> ValueError, match=r"artifacts not pinned to expected revision.*calibration_report", ): - loader._resolve_certified_release( - resolve("uk", 2025, variant="national"), hub_download=hub - ) + loader._resolve_certified_release(resolve("uk", 2025), hub_download=hub) def test_local_area_line_selects_its_only_microdata_artifact(tmp_path: Path) -> None: @@ -568,9 +565,7 @@ def test_loader_refuses_an_unknown_dataset_role(tmp_path: Path) -> None: ) with pytest.raises(ValueError, match="unknown dataset_role 'research_preview'"): - loader._resolve_certified_release( - resolve("uk", 2025, variant="national"), hub_download=hub - ) + loader._resolve_certified_release(resolve("uk", 2025), hub_download=hub) def test_download_uses_pointer_manifest_and_release_revision( @@ -804,7 +799,7 @@ def _is_missing_pointer_error(exc: BaseException, pointer_path: str) -> bool: [ ("us", 2024, DEFAULT_VARIANT), ("uk", 2023, DEFAULT_VARIANT), - ("uk", 2025, "national"), + ("uk", 2025, DEFAULT_VARIANT), ], ) def test_live_latest_pointer_resolves_to_a_coherent_certified_release( @@ -823,6 +818,9 @@ def test_live_latest_pointer_resolves_to_a_coherent_certified_release( which are updated atomically with each certification PR. """ spec = resolve(country, year, variant=variant) + # The UK 2025 default follows its line's pointer since microcosm#823, so + # the pointer, not the variant, says how the release is named. + line = loader._line_from_pointer_path(spec.pointer_path) try: certified = loader._resolve_certified_release(spec) except Exception as exc: @@ -834,26 +832,26 @@ def test_live_latest_pointer_resolves_to_a_coherent_certified_release( pytest.skip( f"repo requires credentials this environment lacks (private): {exc}" ) - if variant != DEFAULT_VARIANT and _is_missing_pointer_error( - exc, spec.pointer_path - ): + if line is not None and _is_missing_pointer_error(exc, spec.pointer_path): # A line pointer exists only once its first cut is promoted; # until then the line is registered but not yet live. Anything # else on a promoted line (a manifest 404, a bad pin) fails. pytest.skip(f"line pointer {spec.pointer_path} not yet promoted: {exc}") raise - assert certified.release_id.startswith( - # Live certified releases still carry the pre-rename id prefix; new - # builds may adopt the microcosm- prefix at the next versioned release. - (f"microcosm-{country}-{spec.year}-", f"populace-{country}-{spec.year}-") - ) assert certified.artifact_repo_id == spec.hf_repo assert certified.artifact_path == spec.filename - if variant == DEFAULT_VARIANT: + if line is None: + assert certified.release_id.startswith( + # Live certified releases still carry the pre-rename id prefix; new + # builds may adopt the microcosm- prefix at the next versioned + # release. + (f"microcosm-{country}-{spec.year}-", f"populace-{country}-{spec.year}-") + ) assert certified.artifact_revision == certified.release_id else: # A line pointer names the immutable cut tag of its constant id. + assert certified.release_id.startswith(f"microcosm-{country}-") assert certified.artifact_revision.startswith(f"{certified.release_id}-") assert re.fullmatch(r"[0-9a-f]{64}", certified.artifact_sha256) assert certified.model.name == spec.engine_package @@ -862,7 +860,7 @@ def test_live_latest_pointer_resolves_to_a_coherent_certified_release( def test_line_pointer_refuses_a_manifest_with_another_lines_role(tmp_path: Path): """A line pointer carries one role: the national pointer cannot select a local-area manifest, nor a local-area pointer a national one.""" - spec = resolve("uk", 2025, variant="national") + spec = resolve("uk", 2025) manifest = _uk_release_manifest() manifest["dataset_role"] = "non_default_local_area" manifest["default_datasets"] = {} diff --git a/packages/microcosm-data/tests/engine_free/shared/test_registry.py b/packages/microcosm-data/tests/engine_free/shared/test_registry.py index eae3a397a5..066a1933ff 100644 --- a/packages/microcosm-data/tests/engine_free/shared/test_registry.py +++ b/packages/microcosm-data/tests/engine_free/shared/test_registry.py @@ -69,8 +69,8 @@ def test_available_lists_default_variant_keys_only_sorted(monkeypatch) -> None: keys = available() assert ("us", 2024) in keys assert ("uk", 2023) in keys - # The national line is off the default variant until promoted. - assert ("uk", 2025) not in keys + # The national line is the default variant since its promotion (#823). + assert ("uk", 2025) in keys assert ("zz", 2099) not in keys assert keys == sorted(keys) @@ -146,16 +146,16 @@ def test_uk_2025_dense_is_registered_off_the_default_variant() -> None: assert spec.filename == "microcosm_uk_2024_25_dense.h5" assert spec.engine_class == "UKSingleYearDataset" assert spec.engine_package == "policyengine-uk" - assert ("uk", 2025, "national") in REGISTRY + assert ("uk", 2025, DEFAULT_VARIANT) in REGISTRY -def test_uk_2025_national_is_registered_off_the_default_variant_until_promoted(): - """resolve("uk") must not reach a pointer that does not exist yet: the - line's pointer appears on the Hub at its first promotion, and only the - follow-up flip to the default variant moves the default (review of - #966; Max's #823 ruling keys the line as a variant).""" - spec = REGISTRY[("uk", 2025, "national")] - assert spec.variant == "national" != DEFAULT_VARIANT +def test_uk_2025_national_line_is_the_default_variant_after_promotion(): + """microcosm#823: the line's first certified cut was promoted on + 2026-10-26, so the line moved to the default variant (Max's #823 + ruling) and its own variant key retired; the June 2023 line stays + addressable at its year.""" + spec = REGISTRY[("uk", 2025, DEFAULT_VARIANT)] assert spec.pointer_path == "latest-national.json" assert spec.filename == "microcosm_uk_2024_25.h5" - assert ("uk", 2025, DEFAULT_VARIANT) not in REGISTRY + assert ("uk", 2025, "national") not in REGISTRY + assert REGISTRY[("uk", 2023, DEFAULT_VARIANT)].filename == "populace_uk_2023.h5" From 45df16363a5db41f927cfc4149092b847e80ebb5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Mar=C3=ADa=20Juaristi?= <127882282+juaristi22@users.noreply.github.com> Date: Sat, 31 Oct 2026 10:21:25 +0000 Subject: [PATCH 2/2] Drop the registry fragment's sentence that the flip makes stale 823-uk-2025-registry.changed.md ended by saying the UK default stays on the June 2023 line until the national line's promotion and the flip of its entry. This PR makes that flip, and both fragments ship in the same release notes (Vahid's review of #1422, round 2). Co-Authored-By: Claude Opus 5.5 --- changelog.d/823-uk-2025-registry.changed.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/changelog.d/823-uk-2025-registry.changed.md b/changelog.d/823-uk-2025-registry.changed.md index 2c9cc54ef5..99c4be3a94 100644 --- a/changelog.d/823-uk-2025-registry.changed.md +++ b/changelog.d/823-uk-2025-registry.changed.md @@ -1 +1 @@ -Registered the UK 2024-25 national line in the data registry as variant `national` on `microcosm_uk_2024_25.h5`, followed through `latest-national.json`, and taught the certified loader to follow per-line pointers at immutable revisions; non-default local-area manifests load only through their line pointers. `available()` now lists default-variant `(country, year)` pairs only (`available_variants()` lists every registered variant, the dense line included). The UK default stays on the June 2023 line until the national line's first promotion and the follow-up flip of its entry to the default variant. +Registered the UK 2024-25 national line in the data registry as variant `national` on `microcosm_uk_2024_25.h5`, followed through `latest-national.json`, and taught the certified loader to follow per-line pointers at immutable revisions; non-default local-area manifests load only through their line pointers. `available()` now lists default-variant `(country, year)` pairs only (`available_variants()` lists every registered variant, the dense line included).