[anth-volk]

Fixes #252

Base: main. This was opened on #247 (head 65ad119), which was stacked on #250. Both merged on 2026-10-21 (#250 at 16:51 UTC, #247 at 19:05 UTC, after #247 was restacked to 3e205e0), and GitHub retargeted this PR to main. Its two commits were rebased onto main (9ee43ff) with git rebase --onto origin/main 65ad119; main's tree is identical to 65ad119's, and git range-diff marks both commits identical (1a66fa7 → d592474, b83108a → 197762f). The counts below were rerun on the rebased commits.

#255 (reading diagnostics schema 9) is stacked on this branch. Head: 0f8321b.

Problem

Release reconciliation threw at the first release it could not publish or repair. It printed no report, never reached the releases after that one, and never checked or updated the latest-release selection. Every later run stopped at the same release. Since 2026-10-02 this has blocked every US release reconciliation (runs 37036146264 and 39488424340), first at populace-us-2024-0fb05b6-4b57d15a287c-20260930T150401Z, so populace-us-2024-buildo-acs-local-766b7c7-20261017T030317Z was never reached. #247 removes that particular error; this PR removes the stop-at-first-error behaviour.

Change

reconcileReleaseBuilds (frontend/packages/calibration-tree/src/publish-calibration-tree.ts) now works the way reconcileStagingBuilds has since #237:

• Each release in its own try. An error for one release becomes that release's failed outcome, with the error as reason and the existing entry's build ID (or null), and the loop continues. This covers every per-release error: a tag that moved ("Released dataset … changed from … to …"), a moved folder, a corrupt stored build, a diagnostics file the reader refuses, a refused repair ("Rebuilt calibration index differs from immutable manifest entry …"), a size limit, an upload error, and an audit that finds the upload incomplete.
• The report is always printed. runPublisher already printed the report and then threw reconciliationFailureMessage(report) for both modes; for releases that message was never reached. It now reads "N of M releases failed to reconcile: . The printed report gives each reason." The staging message is unchanged ("… staging sources failed …"). The process exits with status 1, so the workflow run fails.
• Errors before the loop still stop reconciliation: reading the manifest, listing releases and tags, resolving tag folders, and the first Blob listing. Some of these can be attributed to one tag or folder (an invalid release_manifest.json on a tag, a non-404 error from a folder probe, a folder whose commit can't be found); they still stop every release with no report, as before this PR. See "Review" below.

A failed release leaves no partial state

• No manifest entry. The release steps that write are now: upload the bundle (publishRelease), list the stored files, audit the build, then write the manifest entry. The audit used to come after the manifest write, so an upload the audit found incomplete was already registered. The audit (auditCalibrationTreeBuild) reads the build's own index and the Blob listing, never the manifest, so it gives the same answer before the write. Every error is now raised before the manifest names a new build. The one exception is the manifest write's own read-back check ("Calibration tree manifest verification failed."), which runs after the write was stored; that was already the case and is unchanged.
• No latest-release selection. The selection step (now selectLatestRelease) skips a release that failed in this run and leaves latestReleaseBuildArtifactId naming the build it named before. Without this, a release whose repair failed but which already had a manifest entry would have been selected.
• No index for a new build that fails before the upload. Almost every refusal (reader, digest, size limit, moved tag) happens before any upload. The bundle uploads its index last (calibration-tree-bundle.ts orders it last, uploadCalibrationTreeBundle uploads in order), so an upload that fails part-way leaves no index. An upload that completes but fails its audit leaves its files, index included, under the build's ID with no manifest entry pointing at them. The build ID is a digest of the country, the release ID and the source files' digests (calibrationTreeBuildArtifactId), so the next run rebuilds the same ID, and uploadCalibrationTreeFile compares each existing file byte for byte before reusing it. Nothing reads files that no manifest entry names.
• The latest-release step reports instead of throwing. If the pointer cannot be read, names a release that is not completely published, or names a release that failed in this run, the report gets a new optional field, latestReleaseFailure, with the reason, and the publisher exits with an error after printing the report. It used to throw "Latest release … is not completely published." with no report. The field is absent when the selection was updated, so a successful run prints the same report as before.

Manifest writes and concurrency

• Only the publisher writes calibration-trees/manifest.json (updateCalibrationTreeManifest; the dashboard's lazy comparison writes use other paths). Each write reads the stored manifest, adds one entry and writes with ifMatch on the ETag it read; after a conflict it reads again and writes once more without the condition. A failed release now makes no manifest write at all, so the reads and writes for the releases after it are exactly those of a run without it. The local manifest copy is replaced only by a successful write's return value, and each later write re-reads the stored manifest anyway.
• The workflow's calibration-tree-manifest-publication concurrency group still allows one publisher job at a time across all countries and event kinds, with cancel-in-progress: false. The workflow is unchanged.
• A run with a failure now continues to build the remaining releases, so it holds the concurrency group for as long as a run in which nothing failed, plus the time spent on the failed release. That is never longer than today's run for the same releases once the failure is fixed. timeout-minutes: 30 is unchanged; for reference, the largest recorded US tree took about 11 to 12 minutes to compile on a laptop (see the Group calibration tree geographies by geography ID, not label #250 review).

Dry run

--dry-run records failures the same way, uploads and writes nothing, and only reads the upstream pointer in the latest-release step (as before). It exits with an error if any release failed, after printing the report.

Byte identity of successful publications

Nothing in the bundle, the uploaded files or the manifest entry changed. Only the order of the listing and audit relative to the manifest write changed, and a successful run makes the same uploads and the same manifest writes (same entries, same makeLatest calls). Checked by:

• publish-calibration-tree-golden.test.ts (7 pass): dry-run repairs of the recorded US, UK and Belgium releases still rebuild their recorded index digests, and the staging tests are unchanged;
• calibration-tree-golden.test.ts (12 pass) and test-support/calibration-tree-golden-cases.test.ts (2 pass), with no golden record changed;
• the existing test in publish-calibration-tree.test.ts that publishes tagged UK builds from their shared release folder, which checks every manifest entry and the latest-release selection, passes unchanged.

Tests (red, then green)

Commit	What	Result
d592474 (was 1a66fa7)	Tests only. Five new tests in publish-calibration-tree.test.ts; the in-memory store there now records uploads, listings, audits and manifest writes in order. In publish-calibration-tree-golden.test.ts, the refused-repair test now expects a failed outcome instead of a rejected call.	6 fail, 21 pass across the two files. Each failure is the call rejecting with the failing release's error: "Schema 8 category provider.category references inconsistent providers: provider and other." (4 tests), "Release build microcosm-uk-2024-25-dense-… remains incomplete: missing …/tier-1.json.gz" (1), "Rebuilt calibration index differs from immutable manifest entry cf259cfc…" (1). bun run lint fails only because the tests read the new latestReleaseFailure field (TS2339, two places).
197762f (was b83108a)	The fix and the documentation; tests unchanged	27 pass, 0 fail; bun run lint passes
0f8321b	Comments and one test name only: the new tests now say "tagged build" and "pointer file" for the fixture's release tags and latest-national.json	20 pass in publish-calibration-tree.test.ts; bun run lint passes

The new tests:

1. A UK repository whose releases are tagged builds of shared release folders; the dense tagged build (first in order) has a file the schema 8 reader refuses: it is failed with the reader's error and no build ID; the national tagged build is still published and becomes the latest release; nothing is uploaded or written for the dense one.
2. The dense tagged build's upload audits as incomplete: it is failed and never written to the manifest; the recorded operation order is upload, list, audit, then manifest write for each release.
3. The release the pointer file (latest-national.json) names fails: the selection keeps naming the June release, and latestReleaseFailure says why.
4. runPublisher with two failures (a refused file and a tag that moved to another commit): it prints one report with all five outcomes, then rejects with "2 of 5 releases failed to reconcile: , . The printed report gives each reason."
5. A dry run records the failure and uploads and writes nothing.

At 197762f:

• calibration-tree package (bun test packages/calibration-tree): 378 pass, 0 fail, 34 files (373 on main, plus 5 new);
• publish-calibration-tree.test.ts 20, publish-calibration-tree-golden.test.ts 7, calibration-tree-golden.test.ts 12, test-support/calibration-tree-golden-cases.test.ts 2, package-boundary.test.ts 9, calibration-tree-artifact.test.ts 16, calibration-tree.test.ts 41, all 0 fail;
• app/api/hf-webhook/route.test.ts and lib/microcosm/hf-webhook-config.test.ts (the webhook that dispatches this workflow): 11 pass;
• bun run lint in frontend/ (both tsc --noEmit runs): passes.

Documentation

• docs/calibration-tree-artifacts.md: the release publication section says what a failed release means; "Failure meanings and recovery" now describes a failed release workflow the way it describes a staging one; "Historical reconciliation" no longer says release reconciliation stops at its first release, and describes the order of a release's writes, what files an error leaves, the manifest write's condition, and when the latest-release selection is left unchanged.
• docs/ai/workflows/publish-calibration-tree-artifacts.md: the publish steps include the audit before the manifest entry and the rule that a failed release is never selected; the reconciliation paragraph describes the failed outcome.
• The workflow file never described the stop-at-first-error behaviour, so it is unchanged.

Environment variables

No workflow, deployment script or configuration changes, and the changed code reads no new environment variable. The publisher's existing variables, for reference (names checked with gh secret list / gh variable list, values not read):

Variable	Read by	Provider	Wiring
BLOB_READ_WRITE_TOKEN	runPublisher	GitHub Actions repository secret	job env: of publish-calibration-tree.yml, inherited by bun run
HF_TOKEN (fallback HUGGINGFACE_TOKEN, not set)	Hugging Face reads	GitHub Actions repository secret	job env:
POPULACE_UK_STAGING_HF_TOKEN	UK staging reads (registration token_env)	GitHub Actions repository secret	job env:
CALIBRATION_TREE_BUILD_ENABLED	the job's if:	GitHub Actions repository variable	vars. context
CALIBRATION_TREE_MAX_RAW_BYTES, CALIBRATION_TREE_MAX_GZIP_BYTES	size limits	not set; defaults apply	—

Not changed

• Staging reconciliation, including the order of its manifest write and audit (pinned by Test the written path of staging reconciliation's failed outcome and update its failure documentation #243's test).
• The single-release modes (--latest, --release), which still throw on error.
• No workflow is dispatched by this PR. After it is deployed, the next release webhook or manual dispatch uses the new behaviour.

Review (2026-10-21)

The change is correct (review report kept in the maintainer's notes). Pushed by the review, test first:

Commit	What	Result
73a2c32	Test only: a dry run must report latestReleaseFailure when the release named by the pointer file fails	1 fail, 20 pass (latestReleaseFailure was undefined)
bb77ca6	The check now runs before a dry run returns	28 pass across the two changed test files
f0d1e54	Docs only: the errors that still stop the whole run also include the first Blob listing and the per-tag and per-folder Hugging Face lookups	—

At f0d1e54: the calibration-tree package 379 pass, 0 fail (the counts above are from 0f8321b); golden tests 12, 2 and 7; boundary 9; webhook 11; typecheck passes.

Confirmed: the run still exits 1 when a release fails, and nothing reads the old output; the audit never reads the manifest, so skip and repair decisions and successful publications are unchanged; partial uploads from a failed release are reused safely, because build IDs and index bytes are reproducible and existing files are compared byte for byte; compare-and-swap and the concurrency group are unchanged. Every environment variable is wired: BLOB_READ_WRITE_TOKEN, HF_TOKEN and POPULACE_UK_STAGING_HF_TOKEN are repository secrets passed through the job's env:; CALIBRATION_TREE_BUILD_ENABLED is a repository variable read in the job's if:; the size limits, HUGGINGFACE_TOKEN, MICROCOSM_UK_LOCAL_CALIBRATION_DIR and the repository overrides (POPULACE_HF_REPO, POPULACE_UK_HF_REPO/_REVISION, POPULACE_BE_HF_REPO/_REVISION and the staging ones) have no provider and fall back to defaults, as intended.

Not fixed (follow-ups):

• Medium, already on main: one bad tag or folder still stops every release with no report (see "Change").
• Low: staging still writes its manifest entry before its audit, so an incomplete staging upload gets an entry while a failed release now doesn't.
• Low: a failed new release reports buildArtifactId: null even when its files were uploaded; staging reports the new ID.
• Operational: a run with a failure now builds every remaining release, so it holds the shared concurrency slot longer. The first US run after this merges may compile several trees waiting since 2026-10-02 within timeout-minutes: 30; on a timeout no report is printed, and releases already written keep their entries.

Review round 2026-10-26

Reviewed at f0d1e54, as it would merge into the current main (c84a916). A local merge has no conflicts. On the merged tree, bun test packages/calibration-tree gives 386 pass and 0 fail, and bun run lint and bun run build both pass. main was not merged into this branch, because nothing here needed it.

• The 2026-10-21 fixes are correct and complete.
  • After bb77ca6, a dry run reports latestReleaseFailure when the pointer file names a release that failed. A dry run still does not check whether the named release is completely published.
  • After f0d1e54, the docs list every lookup that still stops the run before the per-release loop.
• main since then. List releases published only as a Hugging Face tag in the release picker #258 adds releases that are published only as a tag to the release list. It reads the dashboard manifest and the folder listing, and it doesn't change the publisher. A failed new release has no manifest entry, so the release list never offers it as a tag-only release. Refuse a country with no configured release instead of serving the US one #248, Start production deploy runs only for main CI runs #261 and Stop trusting the compromised eMudhra CA certificates #256 don't touch the publisher or its workflow. I found no bad interaction with the per-release error handling, the latest-release selection or the dry-run report.
• Findings. Both are low severity and are in comments and docs only, with no change in behaviour.
  • Fixed in 3034a78: the ReconciliationOutcome comment said nothing is written for a failed source. In fact, staging writes its entry before the audit, and a release's uploaded files stay in storage.
  • Fixed in 3034a78 and 64f9cf4: the docs and the loop comment said that a failed release gets no manifest entry and is never selected as the latest release. In fact, an entry the release already had stays, and so does an existing selection. A failed new release does get an entry when the manifest write is stored but its read-back check fails.
• Still open from 2026-10-21. These are the follow-ups listed above:
  • one bad tag or folder still stops every release;
  • staging still writes its manifest entry before the audit;
  • a failed new release still reports buildArtifactId: null;
  • runs are now longer, measured against timeout-minutes: 30.
• The "Dry run" paragraph above is out of date. Since bb77ca6, a dry run also reads the pointer file and reports latestReleaseFailure when the pointer cannot be read or when it names a release that failed in the same run.
• Read calibration diagnostics schema 9, whose hierarchy labels are declared once in catalogs #255: this branch was merged into it in e634386, with no rebase.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
calibration-diagnostics	Ready	Preview	Oct 26, 2026 12:35pm UTC