GHCR node credential rollout fails when a node changes during the rollout #5016

Activity

1. devantler

   added the

   Bug

   issue type on Oct 17, 2026
2. github-project-automation

   added this to 🌊 Project Boardon Oct 17, 2026
3. coderabbitai commented on Oct 17, 2026

   

   coderabbitaiBot

   on Oct 17, 2026 – with coderabbitai

   Contributor

   More actions

   ⚠️ Possible Duplicate Issue(s)

   • Prod deploy evicted from the merge queue when the autoscaler changes a node mid-transaction #3184

   ---

   📝 Issue Planner

   _{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

   • Create Plan

   ---

   🧪 Issue enrichment is currently in open beta.

   You can configure auto-planning by selecting labels in the issue_enrichment configuration.

   To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

   issue_enrichment:
     auto_enrich:
       enabled: false

   💬 Have feedback or questions? Drop into our discord!
4. devantler

   added a parent issue on Oct 17, 2026

   • Production deploys fail only for the change they deploy #4979
5. devantler

   moved this to 📥 Backlog in 🌊 Project Boardon Oct 17, 2026
6. devantler commented on Oct 24, 2026

   

   devantler

   on Oct 24, 2026

   ContributorAuthor

   More actions

   🤖 Generated by the Agentic Engineer (interactive Claude Code session)

   Refinement, 2026-10-24: the first acceptance criterion, which condition failed in each run.

   • 35503104796 and 35505627116 (pr-3927, 2026-09-20 09:43 and 10:38 UTC): an autoscaler node disappeared while the bridge held it, so the identity check refused the patch and the release step then failed. These are the two runs Prod deploy evicted from the merge queue when the autoscaler changes a node mid-transaction #3184 measured on 2026-09-30, and Prod deploy evicted from the merge queue when the autoscaler changes a node mid-transaction #3184 closed as completed on 2026-10-05 with fix(deploy): deselect removed nodes before synchronization #3588 (deselect a Node already gone before mutation), fix(deploy): fence GHCR rollout from autoscaler node deletion #3944 (keep Cluster Autoscaler off a bridge-owned Node) and fix(ci): confirm deleted nodes during fence cleanup #4358 (a confirmed deletion during fence release is treated as nothing left to release).
   • 35524936547 (2026-09-20 17:30 UTC): a different condition. The deploy log reports that an autoscaler node "did not accept the Git/SOPS GHCR registry auth", and the bridge then released its ownership "while preserving the pre-existing cordon" on that node. So the node was already cordoned before the bridge reached it, and its Talos registry patch was not accepted. The log does not say why; it does not show the node being deleted.

   So two of the three runs are the race #3184 fixed. What is left for the second criterion is only the third case: whether a node that is already cordoned when the rollout reaches it (for example one the autoscaler is retiring) should be skipped as expected, or whether a refused registry patch there is a real failure. No change was made.