Repository navigation
fix(ci): wait briefly for the image-verification policy handoff instead of failing the deploy on first contention #2869
Description
🤖 Generated by the Agentic Engineer
Evidence
A production deploy failed today at the image-verification policy handoff gate, and the very next
dispatch of the same workflow on the same commit sailed straight through it:
| Run | Started | Outcome |
|---|---|---|
| 30696866649 | 2026-08-01T11:01:22Z | failure at 11:03:13Z — Another transaction already owns the image-verification policy handoff; refusing cluster mutation. |
| 30696960482 | 2026-08-01T11:04:11Z | passed the same gate (same sha 4cfe7d9f) |
So the contention window that failed the deploy had already closed under a minute later.
Problem
pause_flux_policy_handoff in scripts/refresh-flux-ghcr-auth.sh reads the owner annotation
platform.devantler.tech/ghcr-policy-handoff-owner and, if it is non-empty, returns 1 immediately:
- no wait, no backoff, no bounded retry — the check is a single read, and the enclosing step
fails the job outright; - everything downstream (
Push manifests to GHCR, cosign signing, SBOM/provenance attestation, Flux
reconciliation, Taloscluster update) is skipped, so the deploy is abandoned mid-transaction and
a human must re-dispatch it.
The two mutual-exclusion primitives in this same script do not have the same shape. The GHCR sync
Lease is a real Kubernetes Lease carrying leaseDurationSeconds/renewTime, so it self-expires
and a waiter can reason about it. The handoff owner is an opaque, timestamp-free annotation
("${desired_revision:0:16}-$$-${RANDOM}") with no expiry and no age anywhere in the script's 3481
lines — grep for the annotation returns 4 references, none of them staleness logic.
Refusing to mutate the cluster while another transaction holds the handoff is correct and must
stay. The gap is not the refusal, it is that the guard has zero patience for a condition that is
normal, brief, and self-clearing.
Why it matters
Deploys to production are the delivery path for every other change here. A gate that converts a
sub-minute overlap into a failed deploy plus a manual re-dispatch makes the safe path the annoying
one, which is exactly the pressure that gets guards weakened later.
Related but distinct, and worth noting rather than fixing here: because the annotation has no expiry,
a run killed in a way that skips the cleanup_refresh_work EXIT trap (SIGKILL, runner eviction,
cancellation) leaves the handoff owned with no automatic recovery. The trap covers ordinary failure
exits, so this is a narrow residual rather than today's cause.
Expected behaviour
Wait for the handoff to be released, up to a bounded budget, before refusing.
Acceptance criteria
-
pause_flux_policy_handoffpolls the owner annotation with a bounded wait and backoff instead
of failing on the first observation of a non-empty owner. - The wait budget is explicit and short enough to fail fast when a holder is genuinely stuck.
- On timeout it still refuses with the current error — the fail-closed property is unchanged,
and a test proves a still-held handoff is refused rather than adopted. - A test proves a handoff released partway through the wait is acquired rather than refused
(the case that failed today). - No change to what the guard protects: an owned handoff never yields a cluster mutation.
Size: small — one function, plus its two tests.
Activity
coderabbitai commentedon Aug 1, 2026
🔗 Related PRs
#2633 - fix(deploy): verify incoming image on Talos nodes [merged]
#2635 - fix(deploy): reboot stale nodes so containerd adopts refreshed GHCR auth [merged]
#2793 - feat(cilium): activate homogeneous device selection [merged]
#2813 - fix(platform): harden Cilium exposure and deploy lease [merged]
#2825 - fix(ci): queue prod deploys instead of dropping the pending one [merged]
📝 Issue Planner
Check the box below or use the
@coderabbitai plancommand to generate an implementation plan and prompts that you can use with your favorite coding assistant.🧪 Issue enrichment is currently in open beta.
You can configure auto-planning by selecting labels in the issue_enrichment configuration.
To disable automatic issue enrichment, add the following to your
.coderabbit.yaml:💬 Have feedback or questions? Drop into our discord!
devantler commentedon Aug 1, 2026
The asymmetry is inside the same script, which makes the fix a matter of consistency
Measured at
main,scripts/refresh-flux-ghcr-auth.sh:acquire_sync_lease(GHCR syncLease)for attempt in 1 2 3, with a CAS re-read per attemptSYNC_LEASE_DURATION_SECONDSon a KubernetesLeasepause_flux_policy_handoff(image-verification handoff)return 1"${desired_revision:0:16}-$$-${RANDOM}", no timestampSo the deploy already treats brief contention on its other mutual-exclusion primitive as normal and
waits it out. The handoff is the outlier, and today's failure is exactly the case the sync lease
already handles: run 30696866649
refused at 11:03:13Z, and run 30696960482
passed the same gate a minute later and completed successfully, so
mainis healthy again andthis is a reliability defect rather than an outage.
Reusing the retry shape
acquire_sync_leasealready establishes keeps the fix small and keeps the twoprimitives reasoning the same way, which is worth more than a bespoke wait here.
devantler commentedon Aug 11, 2026
Evidence from today qualifies this issue's premise, so flagging it before the fix is built.
A wait already exists — measured on
#3090today, the gate ran for 3m57s (deploy,reassert_flux_ghcr_auth) and 3m56s (heal,stage_flux_ghcr_auth) before failing. So thecurrent behaviour is no longer "fails on first contention"; it waits ~4 minutes and still fails.
More importantly, today's blocker is not a contention window at all.
flux_policy_handoff_is_quiescent()requires
flux-system/infrastructureto have noReconciling=Truecondition, and that Kustomizationis stuck in a permanent 20-minute
ProgressingWithRetryloop because a Longhorn volume fault leavesCluster/observability/coroot-dbunhealthy. No bounded wait outlasts that — a longer timeout wouldturn a 4-minute failure into a 20-minute one and change nothing.
The 2026-08-01 observation in the issue body (window closed under a minute) still stands for genuine
peer-transaction contention. It just is not the only thing that trips this gate, and the other cause
needs a different fix — distinguishing "parent is mid-apply" from "a workload it owns is persistently
unhealthy".
Full chain: #3092.
devantler commentedon Sep 19, 2026
Closing: the proposed fix can't help, and the real cause has been handled
Waiting for a live peer is impossible now. Every workflow that takes the handoff (the merge-group
deploy-prodandheal-prod-on-failurejobs inci.yaml,cd.yaml, and thedr-rebuild.yamlrebuild job) holds the sharedprod-deployconcurrency group withcancel-in-progress: falseandqueue: max(#2825). No two of them run at once. So an owner annotation that is present when a run starts was left behind by a run that exited without releasing it. That's an orphan, not a peer mid-transaction. Waiting with backoff (this issue's proposed fix) would only turn a quick failure into a slow one.The orphan case now has a proper path. #3034 made the error name
./scripts/refresh-flux-ghcr-auth.sh --fences, which lists every held fence with its liveness evidence and the exact release command. It also added the runbook section Recover an orphaned GHCR deploy fence. The Kustomization-stuck-reconciling case from the 2026-08-11 comment is tracked on #3092.It hasn't happened again. I scanned the logs of all 62 failed platform deploy runs since #3034 merged (2026-08-16): merge-group
ci.yamlpluscd.yaml. None contained this error. Controls: those logs are readable (each carries##[error]lines), and the same pattern does match the original 2026-08-01 run 30696866649.If it comes back,
--fencesoutput from that run is the starting evidence. Reopen with it.