Skip to content

feat(controller): let the module controller authorize its metrics itself - #235

Merged
duckhawk merged 1 commit into
mainfrom
feat/controller-metrics-in-process-authorization
Nov 15, 2026
Merged

duckhawk merged 1 commit into
mainfrom
feat/controller-metrics-in-process-authorization

Conversation

@duckhawk

@duckhawk duckhawk commented Nov 15, 2026 •

Copy link
Copy Markdown
Member

Description

helm_lib_module_controller_manifests gets controllerMetricsInProcessAuthorization (default false). Together with controllerMetricsProxyPort it changes how the controller's metrics are served:

  • the kube-rbac-proxy container and its VPA policy are not rendered;
  • the controller container carries the https-metrics port (controllerMetricsProxyPort);
  • the template emits METRICS_BIND_ADDRESS=:<controllerMetricsProxyPort>, METRICS_AUTHORIZATION_NAMESPACE=d8-<module> and METRICS_AUTHORIZATION_DEPLOYMENT=<fullname>.

The controller is then expected to serve HTTPS on that address and authorize each request the way the proxy did: a TokenReview, then a SubjectAccessReview for get on deployments/prometheus-metrics of its own Deployment.

RBAC is unchanged: helm_lib_module_controller_rbac still binds the controller ServiceAccount to d8:rbac-proxy (TokenReview, SubjectAccessReview), and the module keeps its Prometheus grant on prometheus-metrics. The Service port name https-metrics and the ServiceMonitor stay as they are.

Without the option nothing changes. The chart version goes to 1.72.36; it is independent of #234.

Why do we need it, and what problem does it solve?

Part of reducing the memory requests of the storage modules on master nodes. In every replica of every module controller, a kube-rbac-proxy runs as a separate process with its own VPA request, only to check a bearer token before forwarding /metrics. On a test cluster with three masters that is 55 containers and about 1 GiB of working set. Each one also costs a request of 25Mi on DKP 1.72–1.75, 64Mi on 1.76.0–1.76.11 and about 32Mi on 1.76.12+.

The option keeps the authorization contract of the proxy, so Prometheus needs no new permissions on any DKP version. controller-runtime's own filters.WithAuthenticationAndAuthorization was not used for two reasons. It authorizes the non-resource URL /metrics, a cluster-wide grant that is wider than the per-Deployment prometheus-metrics subresource the proxy checked: any identity allowed to read /metrics of any component could read the controller. It also pulls k8s.io/apiserver into the binary.

The pilot module is csi-nfs.

What is the expected result?

  • A module controller with controllerMetricsInProcessAuthorization: true has no kube-rbac-proxy container. The controller listens on https-metrics and gets the three environment variables above.
  • Modules without the option render exactly as before.

Checklist

  • The code is covered by unit tests.
  • e2e tests passed.
  • Documentation updated according to the changes.
  • Changes were tested in the Kubernetes cluster manually.

@duckhawk
duckhawk marked this pull request as ready for review November 15, 2026 10:51
@kkozoriz
kkozoriz self-requested a review November 15, 2026 10:51
helm_lib_module_controller_manifests put a kube-rbac-proxy container next
to the controller whenever controllerMetricsProxyPort was set. That is one
more process, with its own memory request, in every controller replica on
master nodes, only to check a bearer token before forwarding /metrics.

controllerMetricsInProcessAuthorization (default false) drops the sidecar
and its VPA policy. The controller container then carries the
https-metrics port itself, and the template emits
METRICS_BIND_ADDRESS=:<controllerMetricsProxyPort>,
METRICS_AUTHORIZATION_NAMESPACE and METRICS_AUTHORIZATION_DEPLOYMENT, so
the controller can make the same checks the proxy made: TokenReview, then
SubjectAccessReview for get on deployments/prometheus-metrics of its
Deployment. RBAC stays as it is: the d8:rbac-proxy binding of the
controller ServiceAccount and the Prometheus grant on prometheus-metrics.

Signed-off-by: v.oleynikov <vasily.oleynikov@flant.com>
@duckhawk
duckhawk force-pushed the feat/controller-metrics-in-process-authorization branch from 6b278ac to 8191c70 Compare November 15, 2026 11:00
@duckhawk
duckhawk merged commit 8ce74a0 into main Nov 15, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants