Skip to content

Isolate net-new repo bootstrap CRs so a pending manual bootstrap can't freeze all app delivery #2626

Description

@devantler

🤖 Generated by the Daily AI Engineer
Problem (observed live 2026-07-13). A single net-new repository that cannot self-create froze all app delivery. The github-config Kustomization (wait: true) manages the Crossplane Repository/TeamRepository/DefaultBranch CRs for devantler-tech/kyverno-policies (declared in .github#94). That repo can't be created declaratively — the provider's create POST /orgs/.../repos omits the org's 5 required custom properties (no CRD field), returning 422, so the CR sits InProgress awaiting a one-time org-admin bootstrap. Because github-config uses wait: true, it never reaches Ready; and because the parent apps Kustomization also uses wait: true on its children, apps never reaches Ready either → every deploy-carrying merge_group run fails its apps health check, and renovate/own PRs get evicted from the merge queue. This stayed hidden while the cluster was frozen on an unrelated pull-secret issue (host-audit #8c); the moment that cleared, this became the sole delivery-freeze cause.

Why it matters. Org-config housekeeping (a not-yet-bootstrapped shared repo) should never be able to freeze application delivery. The two concerns are coupled far more tightly than they need to be, turning a benign "pending manual bootstrap" into a portfolio-wide delivery outage.

Proposed direction. Decouple bootstrap-pending, manually-gated repo CRs from the readiness that gates app delivery. Options to evaluate:

  • Place net-new repo CRs whose creation requires a manual org-admin bootstrap in a separate Kustomization with wait: false (or exclude them from health checks) until they are adopted, so a pending bootstrap no longer blocks github-config/apps readiness.
  • Alternatively, relax the apps→github-config readiness coupling so org-config health does not gate app delivery.
    Whichever path, add a guard/test so a future un-bootstrappable repo CR degrades gracefully (surfaced, not delivery-freezing).

Size. M. Part of the resilient-GitOps theme (#2043); relates to the repo-creation bootstrap pattern (#2325) and .github#94.

Note. The immediate freeze self-heals once an org admin bootstraps kyverno-policies (one-click already on .github#94); this issue is the durable fix so the class can't recur.

Activity

  1. coderabbitai commented on Jul 13, 2026

    @coderabbitai
    Contributor
    🔗 Related PRs

    #2319 - feat(security): cosign-verify the infrastructure Flux OCI artifact [merged]
    #2330 - ci: heal prod when a merge-group deploy fails (restore main) [merged]
    #2370 - fix(autoscaling): break the DaemonSet-VPA node ratchet; keep Longhorn storage off autoscale nodes [merged]
    #2378 - fix(policies): split restrict-storage-to-baseline-workers so Kyverno admits it (unfreezes infrastructure) [merged]


    📝 Issue Planner

    Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.

    • Create Plan

    🧪 Issue enrichment is currently in open beta.

    You can configure auto-planning by selecting labels in the issue_enrichment configuration.

    To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

    issue_enrichment:
      auto_enrich:
        enabled: false

    💬 Have feedback or questions? Drop into our discord!

  2. moved this to 📥 Backlog in 🌊 Project Boardon Jul 18, 2026
  3. self-assigned this
    on Oct 4, 2026
  4. added a commit that references this issue on Oct 4, 2026
  5. moved this from 👀 In Review to ✅ Done in 🌊 Project Boardon Oct 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Fields

Priority

None yet

Start date

None yet

Target date

None yet

Effort

None yet

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions