ci: pin promotion policy, published-bytes pin reviewer, release workflow record

HISTORY.md

@@ -41,6 +41,13 @@ pass. No native tests, CI jobs or release artifacts were rebuilt or rerun;
 product sources, workflows, pins and project locks are unchanged. Current scope:
 [CI acceptance](docs/acceptance/ci_acceptance.md#current-hosted-coverage-2026-11-11).
 
+### Add the protected release workflow and the CI pin promotion policy
+
+`release.yml` packages a commit that is on `main` with a passing `CI gate`, verifies the archive on a separate runner (including a real GitHub SDK acquisition and the C and C++ workflows) and publishes only through the `release` environment, which needs a reviewer.
+A design flaw was fixed before the first run (the notes needed a digest that exists only after the build; they are now a token template). A hosted dry run produced a verified draft whose archive, sums file and notes agreed; the draft was deleted, and an unchecked commit
+was rejected at the first job. The real publish step has not been run. The pin promotion policy and `tools/check_ci_pins.py` are in the development guide; every current pin matches the published bytes. The dry run carried an approval record the assistant did not create.
+[CI acceptance](docs/acceptance/ci_acceptance.md#release-workflow-2026-11-11).
+
 ### Add the manual integration workflow
 
 `integration.yml` runs a pin gate, the 54-case restricted-redirect replay and the port suites on Linux x64 and arm64, with every input pinned and size/SHA-256 checked by a unit-tested driver. The redirect replay passes

TODO.md

@@ -65,10 +65,13 @@ paths and the Linux/arm64 dispatch, push and pull-request runs.
 - [ ] `integration.yml` (manual; the 54-case redirect replay passes on Linux x64 and arm64 in hosted run `42957820373`): the five
   port suites still need CRT to publish and pin a per-host independent host-tools bundle and a hosted-runner adaptation of
   `tools/replay_port_linux.py`; until then `pins` and `ports` fail on purpose. Add `schedule:` with the pin; macOS/Windows jobs follow.
-- [ ] `release.yml`: protected, manually approved packaging from a tested CI artifact, independent extraction and
-  GitHub SDK acquisition before publication. CI success is not publication authority.
-- [ ] Define how CI pins are promoted (new Preview release, digest review) and add a Windows/arm64 leg
-  once its public SDK and CLI package exist (the Linux/arm64 leg was added 2026-11-11).
+- [ ] `release.yml` (manual; `package` → `verify` → approval-gated `publish`): the dry run stops at a verified draft and the entry
+  gates reject an unchecked commit ([record](docs/acceptance/ci_acceptance.md#release-workflow-2026-11-11)). Not yet exercised: the
+  real publish step (`dry_run: false`), whose first use should be the next genuine Preview; it needs the reviewer's approval on the
+  `release` environment, and the approval's author is to be recorded with a comment. Linux x64 and arm64 only; macOS and Windows
+  jobs follow the same host-key pattern.
+- [ ] Add a Windows/arm64 leg under the [pin promotion policy](docs/guides/development.md#ci-pin-promotion) once its public SDK, its
+  CLI Preview and a hosted runner exist. The policy and `tools/check_ci_pins.py` are in place (2026-11-11).
 
 ## Planned
 

docs/acceptance/ci_acceptance.md

@@ -487,3 +487,38 @@ source is pinned (`zlib-1.3.1.tar.gz`, 1,512,791 bytes, SHA-256 equal to the rec
 
 Not covered: macOS and Windows integration jobs, the oracle's pinned Python, the port suites themselves, and any repository secret (none is used).
 
+## Release workflow (2026-11-11)
+
+`release.yml` is started by hand (`workflow_dispatch`: host, a full commit SHA, date, serial, a notes template path, `dry_run` defaulting to true). Three jobs, driven by `tools/ci_release.py` (14 unit tests):
+
+| Job | Permission | Does |
+| --- | --- | --- |
+| `package` | read | refuses a commit that is not on `main` or has no successful `CI gate`; builds exactly that commit with the pinned published CLI and public SDK through `ci_bootstrap.py`; assembles the notices; packs twice and requires identical bytes |
+| `verify` | read | on a separate runner: archive digest against its sums file, `package_cli_archive.py verify`, `--version`, self `doctor` with the manifest pin, a wrong pin refused, a real GitHub `sdk acquire`, then C and C++ `new`/`build`/`test`/`run`/warm build |
+| `publish` | `contents: write` | waits for the `release` environment; re-checks the digest, renders the notes template, stages a draft, downloads it and compares byte for byte, then publishes a non-latest prerelease (or stops at the draft) |
+
+The `release` environment requires reviewer `webos21` and deploys from `main` only. `prevent_self_review` is off because a one-person repository could not publish otherwise; the approval is therefore a deliberate pause, not independent review.
+
+**A design flaw found before the first run.** `publish` required the notes to contain the archive SHA-256, which only exists after the build, so a note committed in advance could never pass. The notes are now a reviewed template with explicit tokens
+(`{{ARCHIVE_SHA256}}`, `{{ARCHIVE_BYTES}}`, `{{COMMIT}}`, `{{TAG}}`); a template without the digest token or with an unknown token is refused, and a digest typed by hand does not stand in for the token.
+
+**Local run first.** `ci_release.py package` on the native aarch64 machine produced the archive, with a first attempt (`release-ci-check-r1-failed`) stopping on a bare `StopIteration` because the artifact is named `crt-cli-<host>`; the executable is now located by
+its verified digest and the SDK by the identity the summary records. `verify` passed all steps against that archive.
+
+**Hosted dry run.** Run `42963992535` (`main`, `8063bbf`, `linux-arm64`, serial 9, `dry_run: true`): `package`, `verify` and `publish` succeeded. The draft `v0.1.0-preview.20261111.linux-arm64.9` targeted `8063bbf`, held the 5,036,724-byte archive and the sums file,
+and its notes carried the real digest (`67d2074d...d703`) with no leftover token; the archive, the sums file and the notes agreed after an independent download. The draft was then deleted; no tag was created and no release other than the four published ones remains.
+**An approval record exists for that run** (`approved`, `webos21`) that the assistant did not create; who pressed it and how is not known from the record. Future runs should add an approval comment.
+
+**Rejections.** Run `42965008964` with `6a00885`, which has no `CI gate` check, stopped in `package` ("CI gate on ...: none"); `verify` and `publish` were skipped and nothing was created. Locally, the publish driver also refused a branch name instead of a SHA, notes without the digest,
+a tampered archive and an already-published tag (checked against the real serial-1 release, read only).
+
+Not covered: an actual publication (`dry_run: false`), the other two hosts, the environment's behaviour for a second reviewer, and workflow-level tests for `actions/download-artifact` across a failed upload.
+
+## CI pin promotion (2026-11-11)
+
+[The policy](../guides/development.md#ci-pin-promotion) fixes when a pin may move, what moves with it, the evidence a pull request must quote, what must pass and how a new host is added. `tools/check_ci_pins.py` re-downloads each host's published CLI archive and SDK manifest, archive and
+contract and compares size, SHA-256 and the manifest's listing; it reported `OK` for `linux-x64`, `linux-arm64`, `macos-arm64` and `win-x64` against `.github/ci/pins.json`. Six unit tests make it report a changed bootstrap archive, a size-only change, a changed SDK archive or contract,
+an archive missing from the manifest and a contract digest the manifest disagrees with, so that `OK` is not the only output ever seen.
+
+Not covered: automatic detection that a newer Preview exists, a scheduled run of the checker, and any second reviewer.
+

docs/guides/development.md

@@ -291,8 +291,8 @@ On macOS the Xcode tools come from `xcrun -f`; on Windows pass `--llvm-bin` and,
 not wanted, `--os-sdk`. `--tests` restricts the suites for a quick local pass; CI always runs the pinned set.
 The Python driver is a build-host orchestrator like the other `tools/*.py` replays, not a CLI prerequisite.
 
-Pins change only in a reviewed commit that cites the release record and re-checks the asset digests against the
-published `SHA256SUMS`; never use a moving tag or `latest`. Linux/x86_64 has two required
+Pins change only through a reviewed pull request that follows the [pin promotion policy](#ci-pin-promotion); never use a moving tag
+or `latest`. Linux/x86_64 has two required
 legs, `ubuntu-24.04` and `ubuntu-26.04`, consuming the same repaired Dev SDK pin. A separate required `linux-arm64` leg (`ubuntu-26.04-arm`) consumes the Linux/aarch64 Dev SDK and CLI Preview pins ([CI acceptance](../acceptance/ci_acceptance.md#linuxarm64-leg-2026-11-11)).
 All Linux legs use LLVM 21: noble provisions the explicitly keyed apt.llvm.org repository;
 26.04 uses Ubuntu's archive. Existing macOS/arm64 and Windows/x64 checks remain
@@ -304,6 +304,35 @@ The original `.3` failure remains historical evidence; the producer's TLSDESC
 fix is shipped in `.4`, rather than hidden by a runner upgrade. [TODO](../../TODO.md#github-ci)
 owns integration/release work and any still-pending hosted qualification.
 
+### CI Pin Promotion
+
+`.github/ci/pins.json` (bootstrap CLI and SDK per host) and `.github/ci/integration-pins.json` (integration inputs) are the only
+inputs `ci.yml` and `integration.yml` trust. A pin moves in a pull request, never automatically and never from a branch, tag or `latest`.
+
+1. **When.** Only after the new input is published and its release record exists in [preview acceptance](../acceptance/preview_acceptance.md)
+   or the producer's acceptance. A pin is not raised to a release that is only a draft, a rehearsal or a local candidate.
+2. **What changes.** The release tag, asset name, size and SHA-256 of the bootstrap CLI, and the SDK manifest, archive, contract and
+   content identity together; never one without the others. Do not touch product source in the same pull request.
+3. **Evidence in the pull request.** Run `python tools/check_ci_pins.py` and quote its output: it re-downloads each host's published
+   bytes and fails on a size, digest, manifest or contract mismatch. `OK` for every host is required; a host you did not change must
+   still be `OK`, because a published asset can be withdrawn or replaced. Cite the release record that justifies the move.
+4. **What must pass.** `CI gate` on the pull request (every required leg builds the checked-out sources with the new pins) and, for
+   an SDK change, the integration workflow's redirect replay for the affected hosts. A pin change that only some legs exercise is split
+   by host.
+5. **After merge.** The `push` run on `main` must be green. If the bootstrap CLI changed, record in acceptance that stage 1 and stage 2
+   still match or why they differ; if the SDK changed, record the new suite counts. A failed promotion is reverted, not patched with a
+   retry, and the previous known-good pin stays in history.
+6. **Who approves.** The `main` ruleset requires `CI gate`; the repository administrator can merge. That is a deliberate-pause control for
+   a one-person repository, not independent review. A second reviewer, when one exists, should approve pin changes.
+7. **Adding a host.** A new leg (for example Windows/arm64) needs, in order: a public SDK release for that host with its record, a
+   published CLI Preview for that host built by that SDK alone, a hosted runner label that exists for the repository, and a local
+   `tools/ci_bootstrap.py --host <key>` pass. Only then add the matrix row, the `pins.json` host and the driver's host key in one pull
+   request, and run it through a manual dispatch before treating it as required. A host with no hosted runner is recorded as unsupported,
+   not simulated.
+8. **Release pins are separate.** Publishing a Preview does not move `ci.yml` pins. After publication, raise the pin in its own pull
+   request under this policy, because the published bytes are what the pin must match.
+
+
 The reviewed `src/test/data/zlib.recipe.json` fixture is checked out with LF
 through `.gitattributes` on every host. Recipe verification hashes exact bytes;
 line-ending conversion produces a different, unreviewed input.

tools/check_ci_pins.py

@@ -0,0 +1,92 @@
+#!/usr/bin/env python3
+"""Review aid for .github/ci/pins.json: re-derive every pin from the published release bytes.
+
+A pin is only as good as the check that produced it. This downloads each host's pinned bootstrap CLI archive and the SDK
+manifest, archive and contract from their public releases and compares size and SHA-256 (and the SDK content identity named by
+the manifest) with the pins. It changes nothing and publishes nothing; a mismatch exits 1. Run it in the same pull request that
+changes a pin, and quote its output in the review.
+"""
+import argparse
+import hashlib
+import json
+import ssl
+import sys
+import urllib.request
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parent.parent
+
+
+class PinError(Exception):
+    pass
+
+
+def fetch(url):
+    with urllib.request.urlopen(url, timeout=300, context=ssl.create_default_context()) as response:
+        return response.read()
+
+
+def compare(label, data, size, sha256):
+    actual = hashlib.sha256(data).hexdigest()
+    problems = []
+    if len(data) != size:
+        problems.append('%s: size %d differs from the pin %d' % (label, len(data), size))
+    if actual != sha256:
+        problems.append('%s: SHA-256 %s differs from the pin %s' % (label, actual, sha256))
+    return problems
+
+
+def manifest_asset(manifest, name):
+    for asset in manifest.get('assets', []):
+        if asset.get('name') == name:
+            return asset
+    return None
+
+
+def review_host(host, pins, get=fetch):
+    """Return the list of problems for one host; an empty list means every pin matches the published bytes."""
+    problems = []
+    boot, sdk = pins['bootstrap'], pins['sdk']
+    base = 'https://github.com/webos21/crt-cli/releases/download/%s/' % boot['release']
+    problems += compare('%s bootstrap archive' % host, get(base + boot['asset']), boot['size'], boot['sha256'])
+    manifest_bytes = get(sdk['release_base'] + sdk['manifest']['name'])
+    problems += compare('%s SDK manifest' % host, manifest_bytes, sdk['manifest']['size'], sdk['manifest']['sha256'])
+    manifest = json.loads(manifest_bytes)
+    archive = manifest_asset(manifest, sdk['archive'])
+    if archive is None:
+        return problems + ['%s: the SDK manifest does not list the pinned archive %s' % (host, sdk['archive'])]
+    problems += compare('%s SDK archive' % host, get(sdk['release_base'] + sdk['archive']), archive['size'], archive['sha256'])
+    contract = get(sdk['release_base'] + sdk['contract'])
+    if hashlib.sha256(contract).hexdigest() != sdk['contract_sha256']:
+        problems.append('%s: SDK contract SHA-256 differs from the pin' % host)
+    listed = manifest_asset(manifest, sdk['contract'])
+    if listed is not None and listed.get('sha256') != sdk['contract_sha256']:
+        problems.append('%s: the pinned contract digest differs from the one the SDK manifest lists' % host)
+    for key in ('content_sha256',):
+        content = manifest.get('content_sha256') or manifest.get('sdk', {}).get(key)
+        if content is not None and content != sdk[key]:
+            problems.append('%s: the pinned %s differs from the SDK manifest' % (host, key))
+    return problems
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument('--pins', default=str(ROOT / '.github/ci/pins.json'))
+    parser.add_argument('--host', action='append', help='limit to these hosts (default: all)')
+    args = parser.parse_args()
+    pins = json.loads(Path(args.pins).read_text(encoding='utf-8'))['hosts']
+    failed = False
+    for host in args.host or sorted(pins):
+        try:
+            problems = review_host(host, pins[host])
+        except (OSError, ValueError, KeyError) as error:
+            problems = ['%s: could not be checked: %s' % (host, error)]
+        for problem in problems:
+            print('MISMATCH ' + problem, file=sys.stderr)
+        print('%s %s' % ('FAIL' if problems else 'OK  ', host), flush=True)
+        failed = failed or bool(problems)
+    return 1 if failed else 0
+
+
+if __name__ == '__main__':
+    sys.exit(main())

tools/test_check_ci_pins.py

@@ -0,0 +1,60 @@
+"""The pin reviewer must report a mismatch, not only print OK."""
+import hashlib
+import json
+import unittest
+
+from check_ci_pins import review_host
+
+def sha(data):
+    return hashlib.sha256(data).hexdigest()
+
+BOOT, ARCHIVE, CONTRACT = b'bootstrap-archive', b'sdk-archive', b'sdk-contract'
+
+def world(**changes):
+    manifest = {'assets': [{'name': 'sdk.tar.xz', 'size': len(ARCHIVE), 'sha256': sha(ARCHIVE)},
+                           {'name': 'sdk-toolchain.json', 'size': len(CONTRACT), 'sha256': sha(CONTRACT)}]}
+    manifest.update(changes.pop('manifest', {}))
+    manifest_bytes = json.dumps(manifest).encode()
+    pins = {'bootstrap': {'release': 'v1', 'asset': 'cli.tar.xz', 'size': len(BOOT), 'sha256': sha(BOOT)},
+            'sdk': {'release_base': 'https://example.invalid/sdk/', 'manifest': {'name': 'm.json', 'size': len(manifest_bytes), 'sha256': sha(manifest_bytes)},
+                    'archive': 'sdk.tar.xz', 'contract': 'sdk-toolchain.json', 'contract_sha256': sha(CONTRACT), 'content_sha256': 'c' * 64}}
+    files = {'https://github.com/webos21/crt-cli/releases/download/v1/cli.tar.xz': BOOT,
+             'https://example.invalid/sdk/m.json': manifest_bytes, 'https://example.invalid/sdk/sdk.tar.xz': ARCHIVE,
+             'https://example.invalid/sdk/sdk-toolchain.json': CONTRACT}
+    files.update(changes.get('files', {}))
+    return pins, (lambda url: files[url])
+
+class ReviewTest(unittest.TestCase):
+    def test_matching_pins_report_nothing(self):
+        pins, get = world()
+        self.assertEqual(review_host('h', pins, get), [])
+
+    def test_a_changed_bootstrap_archive_is_reported(self):
+        pins, get = world(files={'https://github.com/webos21/crt-cli/releases/download/v1/cli.tar.xz': b'tampered-bootstrap'})
+        problems = review_host('h', pins, get)
+        self.assertTrue(any('bootstrap archive' in p and 'SHA-256' in p for p in problems), problems)
+
+    def test_a_changed_size_alone_is_reported(self):
+        pins, get = world()
+        pins['bootstrap']['size'] += 1
+        self.assertTrue(any('size' in p for p in review_host('h', pins, get)))
+
+    def test_a_changed_sdk_archive_or_contract_is_reported(self):
+        pins, get = world(files={'https://example.invalid/sdk/sdk.tar.xz': b'other-sdk-archive!'})
+        self.assertTrue(any('SDK archive' in p for p in review_host('h', pins, get)))
+        pins, get = world(files={'https://example.invalid/sdk/sdk-toolchain.json': b'other-contract'})
+        self.assertTrue(any('contract' in p for p in review_host('h', pins, get)))
+
+    def test_an_archive_the_manifest_does_not_list_is_reported(self):
+        pins, get = world()
+        pins['sdk']['archive'] = 'unlisted.tar.xz'
+        self.assertTrue(any('does not list the pinned archive' in p for p in review_host('h', pins, get)))
+
+    def test_a_contract_digest_that_differs_from_the_manifest_is_reported(self):
+        wrong = {'assets': [{'name': 'sdk.tar.xz', 'size': len(ARCHIVE), 'sha256': sha(ARCHIVE)},
+                            {'name': 'sdk-toolchain.json', 'size': len(CONTRACT), 'sha256': '0' * 64}]}
+        pins, get = world(manifest=wrong)
+        self.assertTrue(any('differs from the one the SDK manifest lists' in p for p in review_host('h', pins, get)))
+
+if __name__ == '__main__':
+    unittest.main()