roadmap: decide whether a registry layer (Harbor) earns its place in the supply chain #2894

Activity

1. devantler

   added the

   Epic

   issue type on Aug 2, 2026
2. github-project-automation

   added this to 🌊 Project Boardon Aug 2, 2026
3. coderabbitai commented on Aug 2, 2026

   

   coderabbitaiBot

   on Aug 2, 2026 – with coderabbitai

   Contributor

   More actions

   🔗 Related PRs

   #2625 - fix(deploy): synchronize production GHCR pull auth [merged]
   #2633 - fix(deploy): verify incoming image on Talos nodes [merged]
   #2635 - fix(deploy): reboot stale nodes so containerd adopts refreshed GHCR auth [merged]
   #2793 - feat(cilium): activate homogeneous device selection [merged]
   #2830 - feat(ci): publish Kubescape posture findings to Code Scanning [merged]

   ---

   📝 Issue Planner

   _{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

   • Create Plan

   ---

   🧪 Issue enrichment is currently in open beta.

   You can configure auto-planning by selecting labels in the issue_enrichment configuration.

   To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

   issue_enrichment:
     auto_enrich:
       enabled: false

   💬 Have feedback or questions? Drop into our discord!
4. devantler

   added a parent issue on Aug 2, 2026

   • Roadmap: a resilient, current, well-abstracted GitOps platform #2043
5. devantler

   added a sub-issue on Aug 2, 2026

   • spike: decide the registry-layer question — pull-through cache, tenant RBAC, and the pull-path failure mode #2895
6. devantler

   moved this to 📥 Backlog in 🌊 Project Boardon Aug 2, 2026
7. devantler commented on Sep 23, 2026

   

   devantler

   on Sep 23, 2026 – with Claude

   ContributorAuthor

   More actions

   🤖 Generated by the Agentic Engineer (interactive Claude Code session)

   Decision: decline. No registry layer, neither Harbor nor a pull-through cache, for now.

   This records the outcome of the #2895 spike. The evidence is from main at 140db93 and CI history for 2026-09-09 → 2026-09-22.

   1. The access-control problem is real, but it is not about the registry

   Today there is no registry, mirror or cache: every node pulls straight from upstream. The pain comes from spreading one GHCR credential to many consumers:

   • Talos node auth (talos/cluster/authenticate-ghcr-pulls.yaml)
   • the Flux root artifact
   • OpenBao
   • Kyverno signature reads
   • every tenant namespace

   About 6,500 lines of bridge scripts keep these in step. Rotating the credential reboots nodes, because containerd reads registry auth only at start.

   A project-scoped registry would not cleanly fix the open issues:

   • Scope tenant GHCR pull credentials to authorized packages #2889 (tenant scoping). Harbor could give each tenant a scoped robot account, but only by becoming the sole holder of the GHCR token (see point 3). The per-repository installation token being tested in Prove a repository-scoped installation token can pull that repository's private package #3274 targets the same outcome without new infrastructure.
   • Single source of truth for the ghcr pull credential (OpenBao + ExternalSecret) #2613 (single source of truth). This is about where the credential lives. Harbor would add one more holder rather than remove one.
   • Skip the GHCR-auth reboot for nodes whose containerd already booted with the current credential #2638 (skip needless reboots). Harbor makes these reboots unnecessary only if nodes stop holding a GHCR credential. That is the design point 3 rules out.
   • Align Coroot API-key wiring and registry egress #2685 is already closed.

   2. Registries do not break things today

   • CI: 6,471 runs in the window; 123 failed.
     • Reading the logs of an evenly spread sample of 60 failed jobs found exactly one registry error: a GHCR 404 caused by our own stale tag reference, fixed by fix(security): the signature inventory misreads an image referenced by both tag and digest #4038 (the next scheduled run passed).
     • The error lines of all 249 failed jobs contain no other registry error.
     • There was no Docker Hub rate-limit, timeout or 5xx at all.
     • The real failure causes are our own gates, for example the production-authorization check (52 of 141 root-cause jobs).
   • Cluster: the only record is transient containerd connection resets to ghcr.io / registry-1.docker.io that "succeed on kubelet retry" (the Coroot SLO note in k8s/providers/hetzner/infrastructure/coroot/patches/enable-ha.yaml).
   • A pull-through cache would be justified by a measured failure rate. That rate is currently zero.

   3. Availability and cold start — the deciding point

   • A transparent mirror is safe. Talos falls back to the upstream registry when every mirror endpoint fails, unless skipFallback is set (Talos RegistryMirrorConfig). A cache that is down or not yet started therefore degrades to today's behaviour.

   • Harbor as the credential holder is not safe. Removing the GHCR credential from nodes means private pulls can no longer fall back to upstream. Three things would then depend on Harbor being up:

     • the ksail-operator image, which relies on node auth
     • the Talos first-party signature check, which reads signatures from GHCR with the node credential
     • every tenant's private image

     Harbor in turn needs CloudNativePG, Redis, storage and its own pods running first. Keeping the node credential as a fallback avoids that loop, but then the rotation reboots and the credential spread from point 1 remain.

   • There is no credible cold-start answer that also delivers the benefit, so by the spike's own rule this is a decline.

   4. It would overlap with checks we already have

   • Scanning: Kubescape's vulnerability scanner already builds SBOMs and scans images from every registry. Harbor's Trivy would repeat that work.
   • Signatures: these are already enforced twice:
     • Kyverno verify-app-images (Deny) at admission
     • Talos ImageVerificationConfig at pull
   • Harbor's "prevent unsigned" would be a third gate that could disagree with the other two.
   • Any setup that rewrote image names would also bypass the ghcr.io/devantler-tech/ match rules. (A transparent Talos mirror keeps the names, so this does not apply to it.)

   5. Operating cost

   To run Harbor with high availability, we must supply highly available PostgreSQL, Redis and storage (Harbor HA).

   • Reused from what the platform already runs: CloudNativePG, R2 and Longhorn.
   • New: a shared Redis (today only fleetdm bundles one) and at least two replicas of each Harbor component.
   • Consequence: all of this would sit in the pull path of every workload, on a platform whose recent failures come from its own operations, not from upstream.

   6. Cheaper alternatives, on the same terms

   • Spegel is a stateless peer-to-peer mirror of images already on nodes. It falls back to upstream when no peer has the image, needs no database, and has a Talos guide (Spegel, Talos guide).
     • Would help: re-pulls after node rolls and scale-outs, which the image garbage-collection thresholds (75/70%) make more frequent.
     • Would not help: a cold pull or access control.
     • Caveat: it serves cached private images to anything that can reach its port (Spegel FAQ), so that port would need locking down first.
   • A plain pull-through cache (the distribution registry or zot, behind a Talos mirror with fallback) would absorb Docker Hub limits. The measured Docker Hub failure count is zero.
   • Both are sound designs, but neither has a problem to solve today.

   Revisit when

   • Registry-caused failures become a measurable rate: three or more in a 14-day window, or any Docker Hub rate limit in CI or cluster events. Try Spegel or a fallback-safe cache first.
   • Image pulls are shown to slow node rolls or scale-outs. Try Spegel first, with its port locked down.
   • A real air-gap or data-sovereignty requirement appears.

   This also gives #785 and #822 their missing reasoning; both are cross-referenced to this comment.

   ---

   Generated by Claude Code
8. devantler

   closed this as completedon Sep 23, 2026
9. github-project-automation

   moved this from 📥 Backlog to ✅ Done in 🌊 Project Boardon Sep 23, 2026