Six flux-system controllers fail C-0013 on a missing runAsGroup, blocking the C-0013 narrowing

🤖 Generated by the Agentic Engineer

Evidence

Measured in prod 2026-08-18 on the pre-exception scan surface
(workloadconfigurationscans, whose appliedIgnoreRules is null throughout, so the results are
what a scan sees before any ClusterSecurityException is applied).

All six Flux controllers in flux-system fail C-0013 (non-root containers) on a single missing
field:

Workload	C-0013	C-0016	C-0055	Missing field
`flux-operator`	failed	passed	passed	`securityContext.runAsGroup`
`helm-controller`	failed	passed	passed	`securityContext.runAsGroup`
`kustomize-controller`	failed	passed	passed	`securityContext.runAsGroup`
`notification-controller`	failed	passed	passed	`securityContext.runAsGroup`
`source-controller`	failed	passed	passed	`securityContext.runAsGroup`
`tofu-controller`	failed	passed	passed	`securityContext.runAsGroup`

Kubescape's own suggested fix value is 1000. Each workload already sets runAsNonRoot, which is
why a runAsNonRoot-only check reads these as compliant — C-0013 also requires runAsGroup.

Why it matters now

flux-system is the only namespace excluded from the add-security-context mutation that is
both scanned in-cluster and not covered by another exception. That makes these six the entire
blocker to narrowing C-0013 in pod-security-mutations: narrowing it today would surface six
real findings with no fix behind them, which lowers the compliance score to absorb a gap instead of
closing it (#2824's third acceptance criterion says the floor is re-baselined upward).

C-0016 and C-0055 have no such gap and are narrowed already.

Expected behaviour

The six controllers set runAsGroup in their own spec, C-0013 passes for them genuinely, and
C-0013 then moves from pod-security-mutations-unscoped.yaml into the namespace-scoped
pod-security-mutations.yaml alongside C-0016 and C-0055.

Note on the mechanism

These are Flux components delivered by flux-operator/HelmRelease, so the field cannot be hand-set
on the Deployment — it needs a Flux postRenderers kustomize patch. There are existing precedents
for that pattern in this tree.

Acceptance criteria

All six flux-system controllers set securityContext.runAsGroup in their stored spec.
C-0013 reads passed for all six on the pre-exception scan surface.
C-0013 is moved into the namespace-scoped pod-security-mutations.yaml.
The compliance floor is re-baselined upward, never lowered.

Part of #2824

🔗 Related PRs

#2725 - fix(umami): bootstrap tenant provisioning immediately [merged]
#3109 - fix(deploy): avoid redundant GHCR node reboots [merged]
#3142 - docs(kubescape): correct the controller-rbac migration guidance [merged]
#3212 - docs(kubescape): record the keep decision for the three unfalsifiable controls [merged]
#3214 - security(kubescape): drop all capabilities on the operator's own workloads [merged]

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

Create Plan

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

🤖 Generated by the Agentic Engineer

Criterion 2 is not verifiable today — the fix still is. Split them before starting.

Checked this issue's dependencies while selecting rung-2 work. Three of the four acceptance criteria
are shippable now; criterion 2 is gated on a live blocker that is not named here, and the two
failure modes that follow are both expensive.

Criteria 1, 3, 4 — implementable now. Setting securityContext.runAsGroup on the six
controllers via a postRenderers patch, then moving C-0013 from
pod-security-mutations-unscoped.yaml into the namespace-scoped pod-security-mutations.yaml,
needs nothing external.
Criterion 2 — blocked. "C-0013 reads passed for all six on the pre-exception scan surface"
requires the Kubescape config scan to be producing. It is not: fix(security): a Kubescape framework-load failure still reports zero findings instead of failing #2933 is open and was last
updated today, and pod-security-mutations-unscoped.yaml's own comment says the config scan "is not
currently producing (fix(security): a Kubescape framework-load failure still reports zero findings instead of failing #2933, security(kubescape): a framework-load failure must fail loudly, not read as a zero-finding clean result #3024)" — so the population behind it is "real but presently invisible
to the posture score". (security(kubescape): a framework-load failure must fail loudly, not read as a zero-finding clean result #3024 is closed; fix(security): a Kubescape framework-load failure still reports zero findings instead of failing #2933 is the live one.)

Why this is worth writing down. A run that takes this issue at face value either (a) ships the
patch and then cannot demonstrate criterion 2, and may read the scanner's silence as success —
which is precisely the silent-suppression failure #2447 exists to prevent, and the same
broken-scanner-reads-as-clean trap that motivated #2933 itself; or (b) sees "blocked on the scanner"
and skips the whole issue, leaving a shippable hardening fix unstarted.

Suggested shape: land the runAsGroup patch and the C-0013 move as one slice, and leave
criterion 2 open with its dependency recorded, closing this issue once #2933 restores the scan and
the six read passed. Note that criterion 3 (narrowing C-0013) is what makes a future regression
in flux-system visible — so verifying it later is not optional bookkeeping.

I am not claiming this issue; recording the dependency so whoever does starts from measured state.

🤖 Generated by the Agentic Engineer

⚠️ Implementing this as written would stop every Flux controller from starting

I measured this issue's premise on admin@prod today before picking it up. The gap is real and the six workloads are the right six — but the prescribed remediation (runAsGroup, Kubescape's suggested value 1000) is not safe on these six as they stand, and this repository has already been bitten by exactly this.

The measurement

Workload	pod `runAsUser`	pod `runAsGroup`	pod `runAsNonRoot`	ctr `runAsUser`	ctr `runAsNonRoot`
`flux-operator`	ABSENT	ABSENT	ABSENT	ABSENT	true
`helm-controller`	ABSENT	ABSENT	ABSENT	ABSENT	true
`kustomize-controller`	ABSENT	ABSENT	ABSENT	ABSENT	true
`notification-controller`	ABSENT	ABSENT	ABSENT	ABSENT	true
`source-controller`	ABSENT	ABSENT	ABSENT	ABSENT	true
`tofu-controller`	ABSENT	ABSENT	ABSENT	65532	true

Two things this corrects about the issue body:

runAsNonRoot is set at CONTAINER level, not pod level. The issue's reasoning ("each workload already sets runAsNonRoot") is right, but the field is not where a pod-level patch would sit — so a patch has no pod-level context to extend.
Not one of the six sets runAsUser at pod level, and five set none at container level either.

Why that combination is dangerous

add-security-context.yaml documents the failure mode in its own header, from experience in this cluster:

A pod-level runAsGroup with NO runAsUser yields the OCI user string ":1000", which containerd rejects (user group "1000" is specified without user) so the pod sandbox never starts. Injecting runAsGroup alone did exactly that to homepage (its chart sets runAsUser only at the CONTAINER level) and to tofu-controller's tf-runner.

That is precisely the shape here — and tofu-controller is named in both. A postRenderers patch adding pod-level runAsGroup: 1000 to these six would put source-controller and kustomize-controller into a sandbox that cannot be created, which takes GitOps reconciliation itself down and makes the change hard to revert through the normal path.

What the fix has to do instead

Set runAsUser and runAsGroup together, never runAsGroup alone — the same pairing rule the mutation policy follows for every namespace it does reach.
tofu-controller cannot take 1000. Its container already runs as 65532; pairing a pod-level 1000 against that container's own runAsUser reproduces the mismatch the policy's exemption list exists for (cert-manager / trust-manager / origin-ca-issuer / keda / opencost are excluded for the same reason). Its value must follow its image UID.
Verify each controller actually starts after the patch, before the next one is patched. source-controller and kustomize-controller are the two that can strand the change if they go down.

Also worth knowing before this is scheduled

Acceptance criterion 2 asks that C-0013 reads passed on the pre-exception scan surface. Per the note now in pod-security-mutations-unscoped.yaml, the config scan is not currently producing (#2933, #3024) — so that criterion is unverifiable until those land, and the scan-surface confirmation should be sequenced after them rather than assumed.

I am not claiming this issue — recording the correction so whoever does is not led into the failure the policy header already warns about.

🤖 Generated by the Agentic Engineer

The value is `65532`, not Kubescape's suggested `1000` — which closes the gap the correction above left open

The 09:27Z correction established that runAsGroup must be paired with runAsUser and that
tofu-controller "cannot take 1000 — its value must follow its image UID". That left the issue
un-implementable, because it named no value for the other five. Measured today on admin@prod:

Workload	image	pod `runAsUser`	ctr `runAsUser`
`flux-operator`	`ghcr.io/controlplaneio-fluxcd/flux-operator:v0.50.0`	ABSENT	ABSENT
`helm-controller`	`ghcr.io/fluxcd/helm-controller:v1.5.5`	ABSENT	ABSENT
`kustomize-controller`	`ghcr.io/fluxcd/kustomize-controller:v1.8.5`	ABSENT	ABSENT
`notification-controller`	`ghcr.io/fluxcd/notification-controller:v1.8.4`	ABSENT	ABSENT
`source-controller`	`ghcr.io/fluxcd/source-controller:v1.8.5`	ABSENT	ABSENT
`tofu-controller`	`ghcr.io/flux-iac/tofu-controller:v0.16.5`	ABSENT	65532

Two independent lines of in-repo evidence converge on 65532:

tofu-controller already declares it and starts. It is the same distroless image family as the
other five and is the one controller that names its own UID — 65532. The correction above singled
it out as the exception; the measurement shows it is better read as the reference value.
This repository already treats 65532 as the distroless nonroot identity, with the reasoning
written down. providers/docker/.../coredns/deployment.yaml:128 says so in as many words —
"65532 is the distroless nonroot identity the pinned image already runs as" — and
minio/{deployment,job}.yaml set runAsUser: 65532 / runAsGroup: 65532 / fsGroup: 65532
together, which is exactly the pairing this issue needs.

So Kubescape's suggested 1000 is actively wrong here, in two separate ways: it is unpaired (the
":1000" OCI-user break the correction documents), and even paired it would mismatch the UID these
images actually run as, which is the same class of failure that put cert-manager, trust-manager,
origin-ca-issuer, keda and opencost on the mutation's exemption list.

Remaining check before implementing

The 65532 value is strongly supported but not yet read from the image configs themselves — the
controllers are distroless, so exec-based confirmation is not available (no shell in the image).
Whoever implements should settle it by inspecting the pinned image config, or by patching
notification-controller first and confirming it starts before touching source-controller or
kustomize-controller — those two are the pair that can strand the change, since they are GitOps
reconciliation itself.

Still not claiming this issue; recording the value so the next run does not have to re-derive it.

🤖 Generated by the Agentic Engineer

Measured: there is no single correct value — the six controllers run two different uid/gid pairs

The remedy this issue records (runAsGroup: 1000, Kubescape's own suggestion) is unsafe, and the
correction previously reached for it — "use 65532" — is wrong for four of the six. Both were
inferred; this is measured from the images themselves.

Declared USER in each running image, read from the registry image config on 2026-08-29:

Controller	Image	Declared `USER`
`source-controller`	`ghcr.io/fluxcd/source-controller:v1.8.5`	`65534:65534`
`kustomize-controller`	`ghcr.io/fluxcd/kustomize-controller:v1.8.5`	`65534:65534`
`helm-controller`	`ghcr.io/fluxcd/helm-controller:v1.5.5`	`65534:65534`
`notification-controller`	`ghcr.io/fluxcd/notification-controller:v1.8.4`	`65534:65534`
`flux-operator`	`ghcr.io/controlplaneio-fluxcd/flux-operator:v0.50.0`	`65532:65532`
`tofu-controller`	`ghcr.io/flux-iac/tofu-controller:v0.16.5`	`65532:65532`

The fluxcd/* controllers are 65534 (nobody); flux-operator and tofu-controller come from
different image families and are 65532. A single namespace-wide value is wrong whichever one is
picked.

Why this is a correctness matter and not a preference

Live spec, same date: five of the six set no runAsUser at all, at container or pod level —
only runAsNonRoot: true. The exception is tofu-controller, which already sets
runAsUser: 65532, matching its own image. That is the shape add-security-context.yaml's own
header warns about: adding runAsGroup where no runAsUser is resolvable produces an OCI user
string with an empty uid (":<gid>"), which containerd refuses — so the patch intended to close a
posture gap would instead stop the workload starting.

tofu-controller needing no uid change is the corroboration: it is the one controller that already
carries its image's own uid, and it is the one that would be safe under the naive patch.

The fix these acceptance criteria actually need

Set both fields, per controller, to that controller's own image values — never one uniform pair:

source-controller, kustomize-controller, helm-controller, notification-controller →
runAsUser: 65534, runAsGroup: 65534
flux-operator → runAsUser: 65532, runAsGroup: 65532
tofu-controller → runAsGroup: 65532 (it already has the matching runAsUser)

Delivered as a Flux postRenderers kustomize patch, as the issue's mechanism note says. The
acceptance criteria are unchanged; only the values are.

Sequencing

This change targets kustomize-controller and source-controller — the components any revert would
have to run through — and the cluster's reconciler is currently wedged (#3454). Getting the uid wrong
here removes the way back, which is why the values above are measured rather than adopted from the
scanner's suggestion. Land it only against a healthy reconciler, and verify each controller's pod
restarts and reaches Ready before moving C-0013 into the namespace-scoped policy.

chore: agent-claim #3223 nonce=0361160d5ea1a158a452bb7285359ea0

chore(security): re-approve the rendered authorization surface finger…

🤖 Generated by the Agentic Engineer

Re-measured in prod 2026-08-30 — five of the six are already fixed, and the scan surface no longer discriminates

Two corrections to this issue's evidence, the second of which makes the obvious re-check fail open.

1. Five of the six now set `runAsGroup` genuinely

Read directly from the live Deployment specs (oidc@prod), which is evidence independent of the scanner:

Workload	`runAsUser`	`runAsGroup`
`flux-operator`	65532	65532
`helm-controller`	65534	65534
`kustomize-controller`	65534	65534
`notification-controller`	65534	65534
`source-controller`	65534	65534
`tofu-controller`	65532	none

Those five landed in flux-operator/helm-release.yaml (65532) and flux-instance/flux-instance.yaml (65534, via the part-of=flux label selector). Both carry the runAsUser+runAsGroup pairing this issue's mechanism note calls for. tofu-controller has no runAsGroup at container or pod level — its pod securityContext is {"fsGroup":1337} only.

So the remaining C-0013 gap in flux-system is one workload, not six, and flux-system holds no other Deployment/StatefulSet/DaemonSet.

2. ⚠️ The stated pre-exception surface now applies exceptions — the naive re-check reads `passed` over the real gap

This issue records workloadconfigurationscans as the surface "whose appliedIgnoreRules is null throughout, so the results are what a scan sees before any ClusterSecurityException is applied". That is no longer true. Today every one of the six reads:

status: passed     subStatus: w/exceptions
appliedIgnoreRules: ["pod-security-mutations-unscoped/C-0013"]

All six — including the one that is genuinely broken — report passed. A re-check that reads .spec.controls.C-0013.status.status therefore concludes this issue is done while tofu-controller still has no runAsGroup. The discriminator is subStatus: w/exceptions plus a populated appliedIgnoreRules, not status.

fixPath is not a discriminator either — it is a static remediation hint from the rule definition and is populated identically on all six, including the five that demonstrably have the field set. I checked it as a candidate discriminator and it failed that test.

Negative control for the query path: the same object returns failed for 2 of its 64 controls, so the path resolves and can express a failure. Scanner liveness confirmed separately — newest workloadconfigurationscans in the cluster is 2026-08-30T10:06Z, so this is not the dead-scanner case.

3. What actually blocks the narrowing

tofu-controller is mid-retirement, not a controller awaiting a patch:

its HelmRelease is Ready=False permanently (HelmChart ... does not have an artifact — the retirement pruned the HelmRepository),
it is already absent from this repository, and
stage 1 of the two-stage CRD retirement has completed: terraforms.infra.contrib.fluxcd.io now carries kustomize.toolkit.fluxcd.io/name: infrastructure-controllers, so Flux owns it and stage 2 can prune it safely.

Patching runAsGroup onto it would be a fix applied to a workload scheduled for deletion, through a HelmRelease that cannot reconcile at all. The right sequence is to finish the retirement — which is already tracked as #3480 — and then narrow C-0013.

I am therefore marking this blocked by #3480 rather than patching the sixth controller. Acceptance criteria 1 and 2 are satisfied for five of six; criteria 3 and 4 (move C-0013 into pod-security-mutations.yaml, re-baseline the floor upward) become actionable once #3480 removes the last workload.

🤖 Generated by the Agentic Engineer

Unblocked — the named residual is gone. This record read
devantler-tech/platform#3480 | last-verified 2026-09-01: still blocking, and named exactly one
residual: tofu-controller, whose leftover HelmRelease #3480 was to delete, with C-0013 narrowing by
that deletion rather than by setting a uid on a workload on its way out.

#3480 closed as completed 2026-09-08T15:53:13Z, and the manifest state on main agrees rather than
merely being asserted: the only remaining tofu paths are the retirement artifacts —
k8s/providers/hetzner/infrastructure/controllers/tofu-controller-crd-retirement/ and
scripts/tests/test-tofu-controller-retirement.sh. There is no tofu-controller HelmRelease. The other
five workloads already set the field per this issue's own record (flux-operator at 65532 via its
HelmRelease post-renderer; helm-, kustomize-, notification- and source-controller at 65534 via the
FluxInstance part-of=flux patch).

Removed the blocked label and the stale blocker line.

What I verified and what I did NOT. Verified: #3480 is closed completed, and main carries no
tofu-controller HelmRelease. Not verified: that C-0013 actually passes on the live cluster. That
needs a live posture read, and a Kubescape number must not be trusted casually here — a config scan on
this cluster was once dead for 53 days while every surface read healthy, and a list of
workloadconfigurationscans returns .spec.controls as null, so a list-based sweep reports "no C-0013
finding anywhere" and is simply wrong. Only a single-object get populates controls. So the next run
should confirm the residual is clear with a liveness-first per-object read before narrowing the gate —
and treat a 0 as a suspect scanner until the scan's data age proves otherwise.

The remaining work is therefore this issue's own: confirm the six-workload C-0013 residual is clear on
the live cluster, then narrow the gate so the state cannot regress.

🤖 Generated by the Agentic Engineer

Live read done — C-0013 is clear across the complete flux-system pod-bearing set, so the narrowing
is now actionable. Read-only throughout, liveness-first, per-object (never a LIST).

Scan liveness: LIVE. Last successful config scan 2026-09-08T09:12:36Z with three consecutive daily
successes, and scan objects are still being written (newest cluster-wide object ~31 min before the read).
⚠️ These objects carry no per-object last-scan timestamp and are updated in place, so freshness alone
cannot prove a given verdict was re-derived. It was therefore corroborated by value coherence: every
C-0013=passed verdict was checked against the live pod spec and agrees.

C-0013 per workload — flux-system contains exactly 5 Deployments and no StatefulSet, DaemonSet,
Pod or Job, so this is the complete set rather than a sample:

Workload	C-0013	C-0211
flux-operator	passed	passed
helm-controller	passed	passed
kustomize-controller	passed	passed
notification-controller	passed	passed
source-controller	passed	passed
tofu-controller	no scan object — workload absent	n/a

All five set runAsGroup (65532/65534) and runAsNonRoot=true at container level, pod level unset —
so the pass is real, not a stale artifact. Independent cross-check: C-0013 appears in no deployment's
failed-control set; the controls that do fail there are 2–3 unrelated medium findings per workload,
consistent with the namespace summary and out of scope here.

tofu-controller is genuinely gone from the cluster, not just from main. No Deployment,
StatefulSet, DaemonSet or HelmRelease, and its ServiceAccounts, Role, RoleBinding and cache-encryption
Secret were each individually verified absent. Two residuals a narrowing decision should not inherit
as clean: the CRD terraforms.infra.contrib.fluxcd.io is still installed (CRDs are typically not
pruned), and terraforms CRs are UNKNOWN — listing them is outside the scoped read-only
credential's permissions, so that is genuinely unread rather than "none".

C-0211 passes on all five, with seLinuxOptions absent at pod level and present at container level
— which is consistent with #3465's finding that the control accepts any non-absent value, rather than
contradicting it.

🔴 One thing that changes how future control work should be done, found in passing: the scanner
does not prune scan records for deleted objects — 7 tofu/tf-runner-named records persist whose
live counterparts are all absent. So presence in the scan set is not evidence that a resource
exists. A sweep answering "does workload X still exist?" from scan records alone would have concluded
tofu-controller was still here. Filed separately, since it is a scanner coverage-hygiene gap independent
of C-0013.

chore: agent-claim #3223 nonce=bfc44b35fdc809aa85513cb932d27f4c

Navigation Menu

Repository navigation

Six flux-system controllers fail C-0013 on a missing runAsGroup, blocking the C-0013 narrowing #3223

Evidence

Why it matters now

Expected behaviour

Note on the mechanism

Acceptance criteria

Criterion 2 is not verifiable today — the fix still is. Split them before starting.

⚠️ Implementing this as written would stop every Flux controller from starting

The measurement

Why that combination is dangerous

What the fix has to do instead

Also worth knowing before this is scheduled

The value is `65532`, not Kubescape's suggested `1000` — which closes the gap the correction above left open

Remaining check before implementing

Measured: there is no single correct value — the six controllers run two different uid/gid pairs

Why this is a correctness matter and not a preference

The fix these acceptance criteria actually need

Sequencing

Re-measured in prod 2026-08-30 — five of the six are already fixed, and the scan surface no longer discriminates

1. Five of the six now set `runAsGroup` genuinely

2. ⚠️ The stated pre-exception surface now applies exceptions — the naive re-check reads `passed` over the real gap

3. What actually blocks the narrowing

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Six flux-system controllers fail C-0013 on a missing runAsGroup, blocking the C-0013 narrowing #3223

Description

Evidence

Why it matters now

Expected behaviour

Note on the mechanism

Acceptance criteria

Activity

coderabbitai commented on Aug 18, 2026

devantler commented on Aug 19, 2026

Criterion 2 is not verifiable today — the fix still is. Split them before starting.

devantler commented on Aug 29, 2026

⚠️ Implementing this as written would stop every Flux controller from starting

The measurement

Why that combination is dangerous

What the fix has to do instead

Also worth knowing before this is scheduled

devantler commented on Aug 29, 2026

The value is 65532, not Kubescape's suggested 1000 — which closes the gap the correction above left open

Remaining check before implementing

devantler commented on Aug 29, 2026

Measured: there is no single correct value — the six controllers run two different uid/gid pairs

Why this is a correctness matter and not a preference

The fix these acceptance criteria actually need

Sequencing

devantler commented on Aug 30, 2026

Re-measured in prod 2026-08-30 — five of the six are already fixed, and the scan surface no longer discriminates

1. Five of the six now set runAsGroup genuinely

2. ⚠️ The stated pre-exception surface now applies exceptions — the naive re-check reads passed over the real gap

3. What actually blocks the narrowing

devantler commented on Sep 9, 2026

devantler commented on Sep 9, 2026

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions

The value is `65532`, not Kubescape's suggested `1000` — which closes the gap the correction above left open

1. Five of the six now set `runAsGroup` genuinely

2. ⚠️ The stated pre-exception surface now applies exceptions — the naive re-check reads `passed` over the real gap