diff --git a/changelog.d/uk-cut-modal.added.md b/changelog.d/uk-cut-modal.added.md new file mode 100644 index 0000000000..266cc5d453 --- /dev/null +++ b/changelog.d/uk-cut-modal.added.md @@ -0,0 +1 @@ +`tools/modal_uk_cut.py` runs the UK cut runner in a Modal container, with its work directory on a Modal volume so a rerun resumes, for when the build machine is busy or asleep. diff --git a/changelog.d/uk-cut-pepy-certification.added.md b/changelog.d/uk-cut-pepy-certification.added.md new file mode 100644 index 0000000000..112ce7b17c --- /dev/null +++ b/changelog.d/uk-cut-pepy-certification.added.md @@ -0,0 +1 @@ +`tools/certify_uk_cut_in_policyengine_py.py` certifies a published UK cut into policyengine.py on a fresh branch: it refuses when another pull request is already certifying the cut, checks the regional route before writing anything, runs certify-data, the lock, TRO, constant and snapshot regeneration and the strict bundle check, measures the UC floor and runs the tests, then drafts the pull request with every judgment call left as a checklist. diff --git a/changelog.d/uk-cut-runner.added.md b/changelog.d/uk-cut-runner.added.md new file mode 100644 index 0000000000..82f7057128 --- /dev/null +++ b/changelog.d/uk-cut-runner.added.md @@ -0,0 +1 @@ +`tools/run_uk_cut.py` builds, signs and assembles a Microcosm UK national cut in one resumable command, fetching every input from Hugging Face by the pins the build already enforces (`tools/uk_cut_inputs.json` says only where each input lives). `tools/modal_rebuild_uk_chronicle_artifact.py` rebuilds the pinned Chronicle UK consumer artifact on Modal and hosts it privately. diff --git a/changelog.d/uk-gate-signing-key-registry.added.md b/changelog.d/uk-gate-signing-key-registry.added.md new file mode 100644 index 0000000000..1aa86c968c --- /dev/null +++ b/changelog.d/uk-gate-signing-key-registry.added.md @@ -0,0 +1 @@ +UK gate reports, release-cut certifications and dense gate reports now verify only under a signing key registered in `microcosm.data.gate_signing_keys`, and every UK signer refuses an unregistered or retired key. The national build checks its key before loading any input, instead of failing after the solve. `python -m microcosm.data.gate_signing_keys check` reports whether the key in the environment may sign. diff --git a/conftest.py b/conftest.py index f434d79435..7a7c44ebf3 100644 --- a/conftest.py +++ b/conftest.py @@ -130,3 +130,44 @@ def unexpected_start(*args, **options): yield emitters for emitter in emitters: emitter.close() + + +@pytest.fixture(autouse=True) +def _admit_test_gate_signing_keys(request, monkeypatch): + """Let tests sign with their own UK gate keys. + + Production verifies only keys registered in + ``microcosm.data.gate_signing_keys``. Tests sign with throwaway keys, so + every unregistered key counts as an active test key here, while the + committed entries keep their real status. A test marked + ``committed_gate_key_registry`` sees the committed registry unchanged. + """ + + if request.node.get_closest_marker("committed_gate_key_registry"): + return + try: + from microcosm.data import gate_signing_keys + except ImportError: + return + committed = gate_signing_keys._registry + + class _AdmitTestKeys(dict): + def __missing__(self, key_sha256): + return gate_signing_keys.GateSigningKey( + key_sha256=key_sha256, + label="test-key", + status="active", + custodian="the test suite", + held_in="the test", + registered="test", + evidence="admitted by the root conftest", + ) + + def get(self, key_sha256, default=None): + return self[key_sha256] + + def registry(country): + entries = committed(country) + return None if entries is None else _AdmitTestKeys(entries) + + monkeypatch.setattr(gate_signing_keys, "_registry", registry) diff --git a/docs/agent-guide.md b/docs/agent-guide.md index 10bc99fdda..1d1f685dba 100644 --- a/docs/agent-guide.md +++ b/docs/agent-guide.md @@ -224,7 +224,10 @@ grades every waiver register on the staged frame before the solve; see README "Dry-running the release's registers". Publication also refuses a release whose build recorded staging telemetry that never reached its repo (`--allow-missing-staging` overrides); a build that declared `--no-staging` -publishes without the flag. Never publish or promote artifacts as a side +publishes without the flag. A UK national cut runs end to end with +`tools/run_uk_cut.py` ([uk-cut-anyone.md](uk-cut-anyone.md)): inputs fetched +by pin, a signing key registered in `microcosm.data.gate_signing_keys`, and +an inspection-only `--publish`. Never publish or promote artifacts as a side effect of another task. A UK rowwise run's **staged** bundle (`staged//` in the private repository, written by the build itself) is inspection evidence, not a release: it never moves `releases/` or diff --git a/docs/uk-cut-anyone.md b/docs/uk-cut-anyone.md new file mode 100644 index 0000000000..b6019af68c --- /dev/null +++ b/docs/uk-cut-anyone.md @@ -0,0 +1,346 @@ +# Building a UK national cut without the original builder's machine + +Any team member or agent session with the credentials below can build, sign +and publish a Microcosm UK national cut, then open its policyengine.py +certification pull request. Each input comes from Hugging Face and is checked +against the pins the build itself enforces. + +This is the short path. The step-by-step reference behind it stays in +[uk-national-release-assembly-runbook-806.md](uk-national-release-assembly-runbook-806.md). +The UK hub's runbook of the hand-run build is at +`~/reviews/uk-hub/jobs/microcosm-cut-runbook/RUNBOOK.md` on Max's machine. + +## What you need + +| Need | Where it comes from | Check | +|---|---|---| +| A clean checkout of `main` locked at the policyengine-uk you want | a "Lock policyengine-uk X" PR merged to main (runbook §2) | `tools/run_uk_cut.py preflight` | +| `HF_TOKEN` | write access to `policyengine/populace-uk-private` and `policyengine/populace-uk-staging`, and read access to the input repos below | preflight | +| `MICROCOSM_UK_TERMINAL_GATE_SIGNING_KEY` | a key registered **active** in `microcosm.data.gate_signing_keys` (see [Signing keys](#signing-keys)) | `python -m microcosm.data.gate_signing_keys check` | +| About 60 GB of free disk, AC power, a quiet host | the machine | preflight | + +## One command + +```bash +cd +uv sync --all-packages --locked --extra uk +export HF_TOKEN=… MICROCOSM_UK_TERMINAL_GATE_SIGNING_KEY=… +uv run --no-sync python tools/run_uk_cut.py all --label sat24 # build, certify, assemble +uv run --no-sync python tools/run_uk_cut.py all --label sat24 --publish # and publish for inspection +``` + +Agent sessions on Max's build Mac read the team key with +`agent-secret get microcosm/uk-gate-signing-key-team maxghenis`. The key can +sign once #1350 registers it, after María agrees. `HF_TOKEN` comes from +`agent-secret get HUGGING_FACE_TOKEN_MAX maxghenis`. + +`all` runs these steps in order: + +1. `fetch` +2. `preflight` +3. spine +4. national dry run, then the national calibration +5. re-score, only when the in-build evaluation did not pass +6. certify +7. stage-1 comparison (with `--previous-h5`) +8. assemble +9. publish preflight +10. publish (with `--publish`) + +Every step first checks whether its outputs already exist and verify, and +skips itself if they do. So when a run stops, fix the cause and run the same +command again. + +Nothing is overwritten: + +- An unfinished spine attempt is left for you to inspect. The builder + deletes a spine's sidecars when it starts, including a gate-blocked + attempt's signed report, so a retry needs `--retry-spine`, which moves the + whole attempt to `spine.failed-