Default listener metrics silently drop later metric families under a collector series cap

[jasons42]

Checks

• I've already read the troubleshooting guide and I'm sure my issue is not covered there.
• I am using charts that are officially provided

Controller Version

0.12.1 (defaults verified unchanged through 0.14.2)

Deployment Method

Helm

To Reproduce

1. Enable listener metrics with the default listenerMetrics config (i.e. do not set it):

   metrics:
     controllerManagerAddr: ":8080"
     listenerAddr: ":8080"
     listenerEndpoint: "/metrics"

2. Run two scale sets in the same cluster, scraped by the same collector, where one serves many distinct repositories and job names and the other serves few.
3. Scrape with a collector that has a per-endpoint series cap and truncates in payload order — e.g. the Datadog OpenMetrics check, which defaults to max_returned_metrics: 2000.
4. Query gha_min_runners for each scale set.

Describe the bug

gha_job_execution_duration_seconds and gha_job_startup_duration_seconds default to 45 buckets each, carrying organization, repository, job_name and event_name labels. That is ~47 series per unique label combination, unbounded in practice.

High cardinality on these histograms is already known (#3153, #2739, #3670). The part that does not appear to have been reported: because prometheus/client_golang emits metric families in alphabetical order (internal.NormalizeMetricFamilies does sort.Strings(names), used by Registry.Gather), a collector that truncates in payload order silently drops every metric family sorting after those two histograms. The histograms sit in the middle of the gha_* namespace, so they crowd out unrelated metrics downstream of them.

Observed on two scale sets in one cluster, identical chart config, same collector:

#	Metric family	Busy scale set (~65 job_name x ~18 repository)	Quiet scale set (5 x 3)
1	gha_assigned_jobs	present	present
2	gha_busy_runners	present	present
3	gha_completed_jobs_total	present, accurate	present
4	gha_desired_runners	present	present
5	gha_idle_runners	present	present
6	gha_job_execution_duration_seconds	truncated	present
7	gha_job_startup_duration_seconds	truncated	present
8	gha_max_runners	null	present
9	gha_min_runners	null	present
10	gha_registered_runners	null	present
11	gha_running_jobs	null	present
12	gha_started_jobs_total	null	present

The cut lands exactly on the histogram boundary: everything alphabetically before them reports, everything after is absent. On the busy scale set gha_completed_jobs_total (position 3) is accurate to within 3% of true job volume measured independently from pod events, while gha_started_jobs_total (position 12) reads about 0.3% of it — roughly a 300x undercount. Across a fleet of 11 scale sets, the four with low label cardinality report all twelve families correctly and the seven with high cardinality lose the same five.

We first suspected the listener itself. Ruling that out took a while, and two things made it harder than it should have been:

• The metrics are not merely undercounted, they are absent, so aggregate queries silently describe only the low-cardinality scale sets rather than returning an obvious error.
• gha_min_runners and gha_max_runners are static configuration values. Losing them on a busy scale set while a quiet one in the same cluster reports fine reads as a listener fault, because no amount of job volume should be able to suppress a constant. That is what sent us looking at the listener rather than at cardinality.

Worth noting for anyone else debugging this: container restartCount is not a usable check for "is the listener healthy", because listener pods are created with RestartPolicy: Never (resourcebuilder.go), so a failed listener is deleted and recreated by the controller and restartCount reads 0 regardless.

Suggestions

Cheapest first:

1. Reduce the default bucket count. 45 buckets is high for a duration histogram, and it is the multiplier on every label combination.
2. Drop job_name and repository from the default histogram labels, leaving them available opt-in via listenerMetrics. This preserves the configurability decision from Removes high-cardinality labels from histogram metrics #3556 while making the default safe.
3. Failing either, document in the metrics docs that the two histograms dominate the exposition payload and can crowd out the scale-set gauges under a collector series cap — and that the resulting failure mode is missing metrics, not wrong ones.

Renaming the histograms so they sort last would also work, but that is a breaking change for existing dashboards and is probably not worth it.

Additional context

Related: #3153 (closed, ghalistener cardinality, resolved by making metrics configurable), #3556 (closed, proposed removing high-cardinality histogram labels), #3670 / #3671 (merged, removed runner_id / runner_name from defaults), #2739 (open, same 45-bucket-plus-high-cardinality pattern in the legacy pkg/actionsmetrics server, and notes the Datadog 2000-series default).

Prometheus' own sample_limit rejects the entire scrape rather than truncating, so this specific selective-loss shape needs a collector that truncates in order. The underlying cause — the default config emitting thousands of series per listener — affects both.

[github-actions]

Hello! Thank you for filing an issue.

The maintainers will triage your issue shortly.

In the meantime, please take a look at the troubleshooting guide for bug reports.

If this is a feature request, please review our contribution guidelines.