fix(flux-system): tofu-controller HelmRelease outlived its retirement and is permanently not-ready

🤖 Generated by the Agentic Engineer

Evidence

#3468 retired the tofu-controller, but its HelmRelease outlived the retirement and now reports Ready=False permanently.

Read from the live cluster on 2026-08-30:

flux-system  tofu-controller  72d  False
  HelmChart 'flux-system/flux-system-tofu-controller' is not ready: does not have an artifact

The HelmChart explains why:

failed to get source: HelmRepository.source.toolkit.fluxcd.io "tofu-controller" not found

The retirement pruned the HelmRepository — flux-operator is now the only one left in flux-system — but the HelmRelease carries

kustomize.toolkit.fluxcd.io/prune: disabled

so Flux deliberately does not remove it. It is left pointing at a source that no longer exists, and it cannot ever reconcile again. Its conditions show the split cleanly: Released=True back on 2026-08-13, Ready=False since 2026-08-30T07:33:34Z, when the control plane restarted and re-evaluated it.

Affected audience and impact

No outage: the retired pod is still running from its last successful release, and all 10 Kustomizations are Ready.

The cost is to the health signal. The cluster now sits permanently at 44/45 HelmReleases ready, so "one HelmRelease is not ready" becomes the normal reading. Any check, alert, or survey that asks is anything degraded? now has a standing false positive to look past — and a standing false positive is how a real degradation gets waved through. This portfolio has already had a survey report nothing_on_fire over a genuinely dead prod; a permanently-red resource makes that easier, not harder.

It also leaves the retirement half-done: the workload is still scheduled and consuming resources, and nothing will ever reconcile or update it again.

Expected behaviour

A retired controller leaves nothing behind that reports unhealthy for ever. Either the leftover HelmRelease (and the HelmChart it owns) is removed explicitly, or the retirement keeps a source it can still reconcile against until removal completes.

Because prune: disabled is set on it, Flux will not clean this up — the deletion has to be explicit.

Acceptance criteria

The tofu-controller HelmRelease and its HelmChart no longer exist in flux-system, or reconcile cleanly.
flux-system reports every HelmRelease Ready again, so "not ready" is once more a real signal.
The retired workload's remaining resources are removed too, or an explicit note records why they are kept.
Whatever removes them is applied through GitOps rather than by hand, so a rebuild does not resurrect the state.

Size

Small. The judgement call is whether anything still depends on the retired workload, not the removal itself.

🤖 Generated by the Agentic Engineer

⚠️ The obvious fix would freeze all GitOps delivery — measured, not theorised

This issue is sized "Small", and the natural approach is the one this repo just used for the same retirement: re-add the object to the repository so Flux owns it, reconcile once, then remove it so Flux prunes. That worked for the CRD — terraforms.infra.contrib.fluxcd.io now carries kustomize.toolkit.fluxcd.io/name: infrastructure-controllers, so stage 1 is complete.

It must not be repeated for this HelmRelease. Read from the live cluster 2026-08-30:

kustomization/infrastructure-controllers:
  wait:         true
  timeout:      25m
  healthChecks: <empty>

With wait: true and no explicit healthChecks, Flux health-checks every resource the Kustomization applies. The tofu-controller HelmRelease can never become Ready — the retirement pruned its HelmRepository, so its HelmChart permanently "does not have an artifact". Re-adding it to the repository therefore moves it from unmanaged and not-ready (harmless — the Kustomization currently reports Ready=True, Healthy=True alongside it) to applied and not-ready, which fails the Kustomization health check and takes the full 25-minute timeout on every reconcile.

That is the failure mode already documented in flux-instance/flux-instance.yaml, where the same 25m health-check timeout "FREEZES all GitOps delivery" — and it is a deadlock GitOps cannot self-heal from, because the fix would have to be delivered through the controllers it wedges.

Why the annotation cannot be removed declaratively either

The only thing preventing Flux from garbage-collecting the object is:

metadata.annotations."kustomize.toolkit.fluxcd.io/prune": disabled

The object already carries kustomize.toolkit.fluxcd.io/name: infrastructure-controllers, so Flux does consider it managed — stripping that annotation is sufficient for it to be pruned. But removing an annotation declaratively requires applying the object, which is the same wedge as above. The repo's annotations-transformers component can only add this annotation, never remove it.

Options, with the trade-off stated

One-time out-of-band deletion of the HelmRelease and its HelmChart. AC4 asks for GitOps "so a rebuild does not resurrect the state" — and that property already holds here by a different route: both objects are already absent from this repository, so nothing recreates them. This satisfies AC4's intent at essentially zero risk. It needs a maintainer, since prod access from an unattended run is read-only.
A Kyverno ClusterCleanupPolicy deleting the two objects. Genuinely GitOps-delivered and needs no apply of the target — but it ships a policy whose only job is to delete a named resource, and it then needs its own removal, so it reintroduces a two-stage cleanup to avoid a one-line manual step.
Restore the HelmRepository so the release reconciles cleanly — rejected: that resurrects the retired controller.

Recommendation: option 1. Option 2 is defensible if a hand-run kubectl delete is unacceptable on principle, but it trades a one-off manual action for a permanent-looking policy plus a follow-up to retire it.

This also blocks the C-0013 narrowing

tofu-controller is the only remaining C-0013 gap in flux-system — it has no runAsGroup at container or pod level, while the other five controllers now set it genuinely (65532 for flux-operator, 65534 for the four ghcr.io/fluxcd/* controllers). Removing this workload closes that gap by deletion rather than by patching a controller that is on its way out, which is why #3223 is now marked blocked by this issue.

Not claiming this one — the remedy needs a cluster mutation I cannot perform from an unattended run.

🤖 Generated by the Agentic Engineer

Second instance of this class: `data-product-controller`

Read from prod on 2026-08-31, while reviewing #3476:

NAMESPACE                 NAME                     AGE   READY   STATUS
data-product-controller   data-product-controller   17h   False   could not get Source object:
                                                                  OCIRepository "data-product-controller" not found

Same shape as the tofu-controller case this issue describes: a HelmRelease left pointing at a
source that does not exist, Ready=False, and unremovable because both it and its namespace carry

kustomize.toolkit.fluxcd.io/prune: disabled

It also does not appear in the apps Kustomization's inventory (126 entries, 0 matching; control:
the same query finds 7 HelmReleases by name), which is the consequence of that annotation rather than
a separate defect. apps itself is healthy throughout — Ready=True, prune: true, last applied
17:45:05Z — so nothing is wrong with reconciliation; Flux is doing exactly what the annotation says.

Provenance differs from the tofu-controller case and is worth recording: these objects were created
at 2026-08-30T14:17:35Z by a merge-group deploy attempt of #3476 that was then evicted. 🩹 Heal Prod restored main, but prune: disabled meant the objects it had already applied stayed. So this
class has two entry routes — a completed retirement (tofu-controller) and an evicted candidate
(this one) — and the second produces leftovers for a change that never merged at all.

Why this strengthens the existing impact argument

The issue already makes the health-signal case: a standing Ready=False trains every check to look
past "one HelmRelease is not ready". That is now two standing false positives rather than one, and
the second belongs to an app that does not exist on main — so anyone reading cluster state sees a
component that is not in the repository, failing, indefinitely.

Effect on the acceptance criteria

The existing criteria are written for tofu-controller specifically. This instance is resolvable a
second way that the tofu one is not: #3476 declares the missing OCIRepository, so merging it
adopts these objects and they reconcile rather than needing an explicit delete. Either resolution
is fine; what is not fine is leaving them, and excluding the app from the apps base — one of the
options under discussion on that PR — would strand them permanently.

Worth deciding whether this issue should own both instances or whether the general rule wants its own
item: an evicted merge-group candidate must not leave prune-disabled objects behind for a revision
that never merged. That is the reusable defect; the two HelmReleases are its symptoms.

🤖 Generated by the Agentic Engineer

The impact is larger than "a health signal" — re-measured live 2026-09-01

This issue states "No outage: the retired pod is still running from its last successful release"
and scopes the cost to the health signal. Reading the live cluster today, the residue is wider than
the HelmRelease and HelmChart this issue names:

Resource	State
`HelmRelease/tofu-controller`	`Ready=False` (`SourceNotReady`), 74d
`HelmChart/flux-system-tofu-controller`	`Ready=False`, no source
`Deployment/tofu-controller`	1/1 Running, 74d
`ServiceAccount/tofu-controller`	present
2 × `ClusterRole` + 2 × `ClusterRoleBinding`	present, cluster-scoped

Two things follow that change how this should be prioritised:

The running workload can never be patched. The retirement pruned its HelmRepository, so the
HelmChart has no source and the release path is permanently broken. The pod will therefore keep
running its current image indefinitely — it cannot receive a security update by any route short of
deletion.
It retains broad cluster-scoped RBAC. The bound ClusterRoles include cluster-wide access to
Secrets and Pods, and they are bound to the ServiceAccount the still-running pod uses. That is
standing privilege attached to a component that is retired and unmaintainable.

So this is not only a degraded health reading — it is a live, unpatchable workload holding
cluster-scoped credentials. Suggesting type: Security rather than a tidy-up.

Flux will not resolve this on its own — confirmed

tofu-controller is no longer in the infrastructure-controllers Kustomization inventory
(checked directly against .status.inventory). Flux has forgotten the resource entirely, so the
removal cannot come from re-running or repairing reconciliation; it has to be explicit, exactly as
this issue says.

Root cause is systemic, and is now filed separately

The prune: disabled annotation was never in this workload's manifest. It comes from
bases/components/annotations-transformers/annotations-transformer-production-persistence-prune.yaml,
which applies it to every HelmRelease — measured: 46 of 46 prod HelmReleases carry it,
healthy ones included. tofu-controller is simply the first HelmRelease retired since that component
existed; every future retirement leaves the same residue by construction.

That root cause is filed as #3503. This issue remains the correct place to clean up the existing
instance — the two are complementary, not duplicates.

Note on the remaining acceptance criterion

Criterion 4 asks for removal "through GitOps rather than by hand". Worth recording that this is
harder than it looks for this particular resource: because it is already out of the inventory and
prune-protected, Flux has no path to it, so removal needs either an adopt-then-prune cycle
(re-declare it so Flux re-owns it, then delete it) or an explicitly declared cleanup. A plain
kubectl delete would satisfy the outcome but not that criterion. Not attempted here — prod access
used for this investigation was read-only.

🤖 Generated by the Agentic Engineer

Measured live 2026-09-01 — the obvious fix is the dangerous one

Re-verified from prod before selecting this. Everything the issue describes still holds: the
HelmRelease is Ready=False, its HelmChart reports HelmRepository ... "tofu-controller" not found, and the retired Deployment is still running (1/1, 74d; pod 6d).

Three new measurements change what the fix should be.

1. Nothing owns the object any more — it is not in any inventory

kustomization/infrastructure-controllers .status.inventory → tofu entries: []

Its labels still say kustomize.toolkit.fluxcd.io/name: infrastructure-controllers, but that is a
stale stamp on the object, not ownership. The entry left the inventory when #3468 removed it from
source; prune: disabled blocked the deletion, not the de-inventorying. So there is no Flux
Kustomization that will ever GC it, and no amount of waiting resolves it.

This also means the issue's second option — "or the retirement keeps a source it can still reconcile
against" — no longer applies. There is nothing left to reconcile it.

2. Re-adding it to Git to let prune remove it would risk wedging the controllers layer

The natural read of "remove it declaratively" is adopt-then-prune: re-add the HelmRelease without
prune: disabled, let Flux take ownership, then delete it in a second PR. Do not do that without
solving the health check first. infrastructure-controllers carries:

wait: true
prune: true
force: true
timeout: 3m
retryInterval: 2m

Re-adding puts a permanently-unready HelmRelease back into that Kustomization's inventory, and
wait: true then health-checks it. That is the same class of deadlock flux-instance.yaml already
documents at length for the source-controller replica patch — where a resource that can never become
ready froze GitOps delivery. Today infrastructure-controllers is Ready=True / Healthy=True
("Health check passed in 1.163347647s") precisely because the orphan is outside its inventory.
The current state is inert; the naive fix is not.

spec.suspend: true on the re-added resource is the obvious mitigation, but I could not evidence it:
there are zero suspended HelmReleases in the cluster to observe, so whether Flux's wait skips a
suspended HelmRelease is untested here and should not be assumed on prod.

3. The deletion has no owner, so it needs an explicit deleter

ClusterCleanupPolicy/CleanupPolicy (kyverno.io/v2) are present and kyverno-cleanup-controller
is healthy (2/2, 101d), so a scoped cleanup policy is a genuinely declarative, GitOps-applied deleter
that never touches the Flux inventory and therefore carries none of the risk in (2). Two caveats worth
deciding explicitly rather than discovering later:

No cleanup policy exists in this repository yet, so this introduces a mechanism for one deletion.
Kyverno's cleanup controller needs an aggregated RBAC grant to delete helmreleases. Widening
Kyverno's delete rights over Flux resources is a real security trade-off on a Security-typed issue,
and it outlives the one-time deletion unless it is removed with the policy.

What I could not determine — RBAC, recorded rather than assumed

The issue names the judgement call as "whether anything still depends on the retired workload", and
I could not close it:

terraforms.infra.contrib.fluxcd.io is forbidden:
  User "oidc:ned@devantler.tech" cannot list resource "terraforms" at the cluster scope

Secrets in flux-system are likewise forbidden, so I could not read the stored Helm release either.
That matters because deleting the HelmRelease fires the finalizers.fluxcd.io finalizer, which
triggers a Helm uninstall — which is what removes the retired workload, and is also what would
remove anything still reconciling Terraform CRs. The CRD is deliberately retained in-repo
(controllers/tofu-controller-crd-retirement/), which is consistent with CRs possibly existing.

This is read-only access scope, not a maintainer-authority blocker — it does not stop the work,
it just means the dependency question must be answered by someone whose credentials can list
terraforms, or by an explicit note that none exist.

Recommendation

Answer the terraforms question first — it decides whether the Helm uninstall is safe at all.
Prefer the scoped cleanup policy over adopt-then-prune, and scope it exactly to
kind: HelmRelease, namespace: flux-system, name: tofu-controller.
Treat the Kyverno delete grant as part of the change under review, and remove policy and grant
together once the object is gone.

Not claiming this run: prod was mid-deploy while I measured (flux-system and infrastructure
Kustomizations both carried a transient ghcr-policy-*-owner handoff annotation naming the in-flight
merge-group run 33511058295), so a cluster-affecting change was the wrong thing to start. Claim
retired; the issue is unblocked and startable for the next run with the above settled.

docs(security): restore the sentence and issue reference the rewrite …

🤖 Generated by the Agentic Engineer

Resolved using the issue's recommended one-time exact cleanup after verifying the dependency inventory:

the tofu-controller HelmRelease, HelmChart, Deployment, ServiceAccount, ClusterRoles, and ClusterRoleBindings are absent;
every remaining HelmRelease across the cluster is Ready;
the retired controller is absent from repository desired state, so a reconcile or rebuild cannot recreate it.

The deletion itself was an exact UID-fenced operational action. The durable part remains declarative: there is no tofu-controller desired state to resurrect, and #3503 owns the systemic prune-disabled retirement behavior.

fix(security): update stale C-0211 population rationale after tofu-co…

Navigation Menu

Repository navigation

fix(flux-system): tofu-controller HelmRelease outlived its retirement and is permanently not-ready #3480

Evidence

Affected audience and impact

Expected behaviour

Acceptance criteria

Size

⚠️ The obvious fix would freeze all GitOps delivery — measured, not theorised

Why the annotation cannot be removed declaratively either

Options, with the trade-off stated

This also blocks the C-0013 narrowing

Second instance of this class: `data-product-controller`

Why this strengthens the existing impact argument

Effect on the acceptance criteria

The impact is larger than "a health signal" — re-measured live 2026-09-01

Flux will not resolve this on its own — confirmed

Root cause is systemic, and is now filed separately

Note on the remaining acceptance criterion

Measured live 2026-09-01 — the obvious fix is the dangerous one

1. Nothing owns the object any more — it is not in any inventory

2. Re-adding it to Git to let prune remove it would risk wedging the controllers layer

3. The deletion has no owner, so it needs an explicit deleter

What I could not determine — RBAC, recorded rather than assumed

Recommendation

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

fix(flux-system): tofu-controller HelmRelease outlived its retirement and is permanently not-ready #3480

Description

Evidence

Affected audience and impact

Expected behaviour

Acceptance criteria

Size

Activity

devantler commented on Aug 30, 2026

⚠️ The obvious fix would freeze all GitOps delivery — measured, not theorised

Why the annotation cannot be removed declaratively either

Options, with the trade-off stated

This also blocks the C-0013 narrowing

devantler commented on Aug 31, 2026

Second instance of this class: data-product-controller

Why this strengthens the existing impact argument

Effect on the acceptance criteria

devantler commented on Sep 1, 2026

The impact is larger than "a health signal" — re-measured live 2026-09-01

Flux will not resolve this on its own — confirmed

Root cause is systemic, and is now filed separately

Note on the remaining acceptance criterion

devantler commented on Sep 1, 2026

Measured live 2026-09-01 — the obvious fix is the dangerous one

1. Nothing owns the object any more — it is not in any inventory

2. Re-adding it to Git to let prune remove it would risk wedging the controllers layer

3. The deletion has no owner, so it needs an explicit deleter

What I could not determine — RBAC, recorded rather than assumed

Recommendation

devantler commented on Sep 8, 2026

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions

Second instance of this class: `data-product-controller`