Collect metrics from every inference cluster

[dennis-upbound]

Description of your changes

Implements the metrics half of design/telemetry.md, merged in #363. Today a fleet has no way to get engine, gateway or GPU metrics off its clusters, and the names it would get differ per engine anyway.

Two kinds and a collector.

MetricMapping names the metrics one component emits and what Modelplane calls them — from, to, and fromUnit where the source is measured in something other than what the target name claims. TelemetryDestination names the sinks the fleet's metrics go to: each carries the exporter type to send with, that exporter's own config, and its own secretRef.

compose-serving-stack renders a collector per InferenceCluster: a ConfigMap of the rendered YAML, a Deployment running it, and the RBAC the Prometheus receiver needs for pod discovery. It scrapes engines, the gateway and the substrate, and relabels Modelplane's identity off the pod labels, so a mapping never has to carry labels of its own. Modelplane provides five built-in mappings covering vLLM, SGLang, the gateway, the picker and DCGM; an operator's mapping runs after them, through the same code. The Deployment carries a sha256 of the rendered config, so editing a mapping or a destination restarts the collector and an unchanged one doesn't.

Mappings compile to OTTL here rather than being written as OTTL in the API. Knowing the collector's configuration language is this function's job, not an operator's: the typed fields validate at the apiserver, they don't pin what an operator wrote to one collector's grammar, and they leave room to render something other than OTTL later. fromUnit earns its place immediately — DCGM reports framebuffer memory in MiB and energy in millijoules, both renamed onto names claiming bytes and joules, and a field that asks the question catches the one we'd got wrong.

A sink types what belongs to Modelplane and passes through what belongs to OpenTelemetry. name, type, endpoint, secretRef and auth are fields. Everything else the exporter takes — TLS, retry, queueing, compression — goes under config unread.

auth.bearerTokenKey names the key in the sink's Secret and Modelplane composes the bearertokenauth extension and wires the reference, because the collector carries no credential on an exporter, only a reference to an extension. A scheme Modelplane doesn't compose still works: define the extension under spec.extensions and name it from the sink's config.

The rest stays pass-through because the schema is upstream and moves on its own schedule — TLS alone is 16 keys, and the Prometheus remote-write exporter is deprecating its top-level HTTP settings in this release series. Typing it would mean a Modelplane release per setting. The OpenTelemetry Operator made the same call twice, most recently when it typed its pipeline wiring in v1beta1 and left exporters and extensions as preserve-unknown-fields.

The collector is rendered as a Deployment rather than an OpenTelemetryCollector. That CRD would put the OpenTelemetry Operator on every GPU cluster to own and upgrade, for a Deployment and a ConfigMap, and the receiver does its own service discovery so there's no target allocator to want either.

The mapping and destination functions compose nothing themselves, so both could mark Ready and stop. They don't, because Ready meaning nothing was checked is how a fleet produces no telemetry with every object green. A MetricMapping reports how many clusters took its renames and isn't Ready at zero. A TelemetryDestination catches a sink naming an authenticator no extension defines — the collector refuses to start on that, so otherwise the failure surfaces two layers from its cause.

Docs go under Platform as Monitor the Fleet, replacing the Prometheus-stack guide, which is deleted.

Scope

The design describes a surface of 50 metrics. This composes the 23 reachable by renaming one metric to one name and converting its unit: the engine, GPU and gateway-latency core. #476 enumerates the other 27 grouped by what blocks each — 11 need per-object state from resource-state-metrics, 8 need MetricMapping to grow the label-folding and histogram-count forms, 2 need the collector to merge replicas twice, and 6 need instrumentation that doesn't emit yet. The guide describes what this composes, not the whole design.

Verification

Six bugs in this path reached a green CI before any of this was run: unit tests,
crossplane render and the collector's own config validation all pass without
scraping anything. So each layer below was run against something that does.

The collector's own validator

otelcol validate inside otel/opentelemetry-collector-contrib:0.161.0, against the
config this composes. Three sink topologies: one sink no auth, two sinks with per-sink
credentials, and two sinks of one type. All exit 0.

Caught: a value rewrite rendered into the metric context, where value_double is not a
valid path and the collector refuses to start.

Local, two clusters, no cloud or GPU (nix run .#e2e -- --verify)

Now asserted in CI. The mock engine serves /metrics with two real names, and --verify
checks they arrive renamed, converted, and attributed:

Asserted
vllm:num_requests_waiting	→ modelplane_requests_waiting
DCGM_FI_DEV_FB_USED at 1024 MiB	→ modelplane_gpu_memory_used_bytes at 1073741824
identity off the pod	cluster, namespace, deployment, engine, role
vllm: names downstream	absent — nothing unrenamed leaves a cluster

Caught: the engine scrape job matched a label value that never existed, so engine metrics
were never collected at all; the engine's container port was unnamed, so the job would have
matched nothing even once the label was fixed; the gateway job asked Envoy for /metrics
and got a 404 every interval; the substrate job ignored prometheus.io/port beside the
annotation it read, and scraped cert-manager's TLS webhook; and every series was labelled
with the ServingStack's generated name rather than the cluster's.

Real GKE, real L4

A source: GKE InferenceCluster provisioned into a live project: VPC, subnet, service
account and key, ProjectIAMMember, the cluster, and both node pools (g2-standard-8 GPU,
e2-standard-4 system). READY=True, serving stack installed, collector 1/1 Running,
zero restarts, zero scrape failures, zero deprecation warnings.

The seven GPU series arrive from the card, carrying the node and its UUID, and the
conversions hold on live values:

Metric	Value
modelplane_energy_joules_total	7060.8	DCGM's millijoules ÷ 1000
modelplane_gpu_power_watts	16.89	idle L4
modelplane_gpu_temperature_celsius	42	idle
modelplane_gpu_memory_used_bytes	0	nothing loaded

Caught: every modelplane_gpu_* series and the energy counter were empty on GKE. Nothing
annotates DCGM for scraping — GKE runs a managed exporter carrying only its own component
labels, the NVIDIA GPU operator installs none there, and Modelplane installs none either.

Real vLLM on that cluster

vllm/vllm-openai:v0.23.0 serving Qwen2.5-0.5B-Instruct on the L4, with traffic through it.
Twenty series arrive renamed — thirteen from the engine, seven from the card — each carrying
cluster=gke-us-central, deployment=qwen-demo, engine=qwen, role=Standalone.

modelplane_gpu_memory_used_bytes reads 22215131136, which is 21186 MiB × 1048576: vLLM
holding ~92% of the L4's 23034 MiB, its default gpu_memory_utilization. The conversion is
exact on a real loaded model rather than a synthetic value.

A Grafana dashboard over that Prometheus

A prometheusremotewrite sink into the cluster's own Prometheus, with panels over
modelplane_requests_waiting, modelplane_kv_cache_utilization_ratio, the four GPU stats
and a p99 over modelplane_request_ttft_seconds_bucket. 34 series arrive, grouped by
deployment and engine.

Caught: a Prometheus backend received every series stripped of its identity — no cluster,
deployment, engine or role, only job and instance. The identity is held as
resource attributes, which the remote-write exporter drops unless told otherwise, and the
debug exporter prints resource attributes, so every check before this one showed them
arriving. A panel legend rendering as / is what exposed it.

I have:

• Read and followed Modelplane's contribution process.
• Run nix run .#build and the python, ty-*, test-* and docs-vale checks.
• Added or updated tests.
• Signed off every commit with git commit -s.

🤖 Generated with Claude Code

[github-actions]

Docs preview: https://modelplane-docs-pr-470.vercel.app (ready once the site's Content workflow finishes)

[haarchri]

need to add the repo and write access to our CI team

[haarchri]

need to add the repo and write access to our CI team

[haarchri]

please set a groupId as category or add a new: https://github.com/modelplaneai/modelplane/blob/main/docs/data/apigroups.yaml#L8

[dennis-upbound]

Added platform to both. They were the only two XRDs without a group category.

[haarchri]

please set a groupId as category or add a new: https://github.com/modelplaneai/modelplane/blob/main/docs/data/apigroups.yaml#L8

[dennis-upbound]

Added platform to both. They were the only two XRDs without a group category.

[negz]

Given this introduces new Modelplane APIs, I think it deserves documentation in the 'main' docs - I'm thinking a new 'Monitor the fleet' section (or something like that) under the Platform heading in the sidebar. Ideally we'd add those docs in this PR.

[dennis-upbound]

Moved it to Platform as "Monitor the Fleet" and deleted the Prometheus-stack guide it replaces, in this PR. The two pages that linked there link here now.

[negz]

Can you enable more than one exporter at once? Do we expect other types?

Would an API like this work?

apiVersion: modelplane.ai/v1alpha1
kind: TelemetryDestination
metadata:
  name: platform
spec:
  sinks:
  - name: otel
    protocol: OTLP                # OTLP | PrometheusRemoteWrite
    otlp:
      endpoint: https://otel.acme.example
      transport: HTTP             # HTTP | GRPC
    auth:
      scheme: Bearer              # None | Bearer | Basic
      bearer:
        secretRef:
          name: otel-token
          key: token
  - name: prometheus
    protocol: PrometheusRemoteWrite
    prometheusRemoteWrite:
      endpoint: https://prom.acme.example/api/v1/write
    auth:
      scheme: Basic
      basic:
        secretRef:
          name: prom-credentials
status:
  conditions:
  - type: Ready
    status: "False"
    reason: SecretNotFound
    message: "Sink otel: Secret modelplane-system/otel-token has no key token"

I would imagine potentially allowing multiple TDs like this but rolling them up to form the actual composed config.

The goals of this sketch:

• Use lists of named subobjects, e.g. to allow multiple sinks of the same type
• Associate sinks with their creds
• Use a required discriminator field to clearly discriminate the union

[dennis-upbound]

Took the named list and the per-sink credentials. That second one was a real bug — secretRef was one Secret for the whole object, so two sinks had to share it and tell their keys apart by prefix. Each sink mounts under its own directory now.

Left the body pass-through. The exporter schema is upstream and versioned separately from us, so a closed protocol enum means a Modelplane release per exporter and dropping the tls/retry/sending_queue settings destinations actually need — tls alone is 16 keys, and insecure_skip_verify is what an internal endpoint wants. The OTel Operator hit this twice and passed through both times, most recently typing the pipeline wiring in v1beta1 and leaving exporters/extensions as preserve-unknown-fields.

No discriminator either: Prometheus Operator uses optional siblings validated in the controller, and KEP-1027 never shipped, so there's no APIUnions gate and it'd be hand-written CEL. Say the word if you want one anyway.

[dennis-upbound]

Went further on this — took the middle. A sink now types what's Modelplane's and passes through what's OpenTelemetry's:

sinks:
- name: primary
  type: otlphttp
  endpoint: https://otel.example.internal
  secretRef:
    name: telemetry-credentials
  auth:
    bearerTokenKey: token
  config:            # anything else the exporter takes
    compression: gzip

endpoint is required and typed — every exporter has one and it was the setting most worth catching at the apiserver. auth composes the bearertokenauth extension and wires the reference, since the collector carries no credential on an exporter, only a reference to one; that was previously three hand-written stanzas. Both render after config, so a passed-through key can't redirect a sink or unpick its credential.

One scheme, not an enum of them, per the conventions note about anticipating a union with a single optional field — and bearer reads a file from the sink's own directory, which sidesteps the env-var collision two Secrets would otherwise have. Anything else still works the old way: define the extension under spec.extensions and name it from config.

Still not typing the exporter body. TLS is 16 keys, retry and queue another dozen, and prometheusremotewrite is mid-deprecation of its top-level HTTP settings as of v0.158.0 — typing that means versioning our API through theirs.

[negz]

I know a lot of this feedback is more design shaped. Sorry it's coming in so late.

[negz]

Do we actually provide Grafana dashboards?

[lsviben]

I assumed we would provide them as examples in documentation, we can have them in the repo of course but i dont expect us settinf up grafana ourselves.

[dennis-upbound]

We don't — there's nothing in the repo. Dropped the claim.

[negz]

Not super actionable feedback, but I don't love that we leak OTTL here. It'd be nice to abstract it so we have the freedom to switch out to another metric collector if we wanted to in future.

OTOH the only way I can do that is something like this, which feels like it'd have the downfalls of Crossplane P&T:

apiVersion: modelplane.ai/v1alpha1
kind: MetricMapping
metadata:
  name: my-engine
spec:
  metrics:
  # Rename.
  - from: my_engine_queued_requests
    to: modelplane_requests_waiting

  # Rename and convert units. The target's unit is known, so the mapping
  # says what the source is measured in, not a factor.
  # Real cases: SGLang KV transfer in ms, DCGM energy in mJ, FB_USED in MiB,
  # thermal violation in ns.
  - from: my_engine_kv_transfer_ms
    fromUnit: Milliseconds
    to: modelplane_request_kv_transfer_seconds

  # Fold two metrics into one, told apart by a fixed label.
  # Real case: vLLM prompt and generation tokens become
  # modelplane_tokens_total{direction}.
  - from: my_engine_input_tokens_total
    to: modelplane_tokens_total
    labels:
    - name: direction
      value: input
  - from: my_engine_output_tokens_total
    to: modelplane_tokens_total
    labels:
    - name: direction
      value: output

  # Rename a label and map its values onto Modelplane's vocabulary.
  # Real case: each engine's finish vocabulary becomes one reason label.
  - from: my_engine_requests_finished_total
    to: modelplane_responses_total
    labels:
    - name: reason
      from: finish_reason
      values:
        eos: stop
        max_tokens: length
        cancelled: abort

  # Take a histogram's count as a counter.
  # Real case: the gateway's duration histogram becomes
  # modelplane_requests_total.
  - from: my_engine_request_duration_seconds
    part: Count
    to: modelplane_requests_total

[dennis-upbound]

Did it. spec.metrics is from/to/fromUnit and the function compiles to OTTL. Left out labels and part from your sketch — both real, neither has a caller yet.

fromUnit caught a live bug while I was converting: DCGM reports FB_USED in MiB and we were renaming it to modelplane_gpu_memory_used_bytes with no conversion, so the series read a millionth of the memory in use. A field that asks the question catches that; a hand-written statement doesn't.

[negz]

Consider avoiding bools: https://github.com/kubernetes/community/blob/c591129c16bf6b57b6ef9d689f1f5175ace56634/contributors/devel/sig-architecture/api-conventions.md#primitive-types

[negz]

FWIW I missed the discussion on #363 but I would recommend not adding this at all until we have clear demand for it. iiuc Part of the motivation was around migrations to this new design, which we shouldn't be prioritizing right now at this early stage of the project.

[lsviben]

+1, lets make it a follow up that we pick up when needed

[dennis-upbound]

Gone.

[negz]

Are we sure we need this? I'd rather leave it out and add it once there's clear demand for it.

[dennis-upbound]

Gone. Nothing read it either — there's no fleet collector composed here, each cluster exports straight to the destination's sinks, so a platform already running one points those at its own endpoint. If we do compose one later, whether to is a composition-level choice rather than a field.

[negz]

Ideally I'd like the built-in MMs for vLLM and SGLang to actually show up in the API so you can see them like any other MM, and to avoid needing two separate codepaths. I can't think of any obvious place to compose them though.

The next best thing would be to specify all of this config as two hardcoded MetricMapping Pydantic objects and artificially prepend them to the mappings required resources. That way we exercise the same code whether the user is specifying MMs (which I imagine'll be pretty uncommon) or we're using our built-ins.

[dennis-upbound]

Done — five of them, prepended to the required resources, so built-in and operator mappings go through the same code. They're the same shape now too, since the kind holds from/to rather than statements.

[negz]

This is a pretty old version right?

(Also might be worth asking an agent to review commentary in this PR for low value / filler content. This comment reads a bit filler-y.)

[lsviben]

Agree, we should start with a newer version plus probably add a renovate to update as OTEL stuff release quite often

[dennis-upbound]

Bumped to 0.161.0. 0.162.0 is tagged but has no multi-arch manifest on Docker Hub yet. Did a pass over the commentary in this PR and cut three, that one included. No renovate rule yet — worth a follow-up.

[negz]

Could/should any of this be Manifest entries in the array of common serving stack components?

I assume some of it (e.g. config computed at runtime) can't. The commentary should clarify why.

[dennis-upbound]

Not really — the stack list is fixed at build time and every object here depends on request-time data: the ConfigMap is compiled from the destination and the mappings, and the Deployment carries that config's digest plus the per-sink secret mounts. The ServiceAccount and RBAC would fit, but splitting one component across two mechanisms would put a collector's permissions on clusters running no collector. Put that in the docstring.

[dennis-upbound]

The fleet's telemetry in Grafana, from the GKE cluster above: a real L4 serving Qwen2.5-0.5B, exported by the composed collector through a prometheusremotewrite sink.

GPU memory reads 20.7 GiB because the value is bytes — DCGM reports MiB and the name claims bytes, which is what fromUnit is for. Energy is 123 kJ from DCGM's millijoules.

Taking this caught the last bug. Every legend rendered as /: Prometheus was receiving each series stripped of its cluster, deployment, engine and role. Those are resource attributes, because that's what the replica merge groups on, and the remote-write exporter drops resource attributes unless asked. The debug exporter prints them, so every check before this one looked fine.

(The doubled stats are pre- and post-fix series in the same window.)