55% of job lifetime is queue, and the wait is concentrated on the Docker-capable class

[rldyourmnd]

Reporting as a tenant (NDDev-it-com/almaty-libraries). Not an outage — the fleet placed every job and nothing timed out. It is a shape worth knowing, because the same ratio was measured a fortnight ago on entirely different hardware, which suggests it is a property of how work is distributed rather than of capacity.

Measurement

One full quality-gates wave, run 32813110833, 2026-08-25T05:30Z. Queue is started_at - created_at, work is completed_at - started_at, summed across all jobs:

jobs	28
queued	87 min — median 4.0 min, max 11.3 min
working	71 min — median 2.2 min, max 7.0 min
queue share of job lifetime	55%

Sampling "time since the last placement" would have given a different and wrong answer; this counts every job's own wait.

Where the wait actually is

The four longest queues in the wave are the same four jobs, in order:

11.3 min  Backend pytest shard 4/4
 9.8 min  Backend pytest shard 3/4
 6.4 min  Backend pytest shard 1/4
 4.8 min  Admin staging symbolication build

Those pytest shards are the ones that declare services: sidecars — they need the Docker-capable class. The other 24 jobs in the wave placed in about a minute. So the wave is not waiting on the fleet in general; it is waiting on one class, and it waits there four times per wave because the suite is sharded four ways.

The longest work is Infrastructure contracts shard 1/6 at 7.0 min, which matches what the same shard takes on a developer workstation — so execution time looks healthy and is not where the loss is.

Prior measurement, for comparison

On 2026-08-15, across 149 application jobs, we measured median queue 23 s but p90 1176 s and max 1679 s — 12.3 hours waiting against 8.9 hours working, 58% queue. That was the retired repository runner host. Different hardware, same ratio.

What we are doing on our side

Reducing how many of our jobs need Docker at all: of seventeen quality-gates jobs only three genuinely require it. That is ours to fix and we are not asking for capacity.

What might be useful fleet-side

1. Whether the Docker-capable class's in-flight cap is the binding constraint here, or whether these jobs are queuing behind non-Docker work that could have been placed elsewhere.
2. Whether a wave submitting four sidecar jobs at once is scheduled as four independent admissions or serialized — Serialize saturated capacity probes across scale sets sharing one fleet budget #107 touched serialization of saturated capacity probes across scale sets sharing one budget, and this may be the same budget seen from the tenant side.

Numbers above are reproducible from the run IDs with the tenant token; happy to re-measure on a wave of your choosing.

[rldyourmnd]

Post-fix natural measurement from NDDev-it-com/almaty-libraries main run 32823562365 after PR #953 and during provider .75 rollout. Across 28 jobs: queue sum 4,588s (76.5m), work sum 4,566s (76.1m), queue share 50.1%, queue average 163.9s. This improves the reported 87m/55% baseline by ~12% queued time, and the fleet stayed OOM-free; provider .75 bounded the burst before delayed PSI feedback and reduced observed CPU/IO PSI substantially. The Docker-capable lane remains the binding path: 8 jobs, queue sum 1,907s, work 1,690s, queue share 53.0%, average wait 238.4s; pytest shards 3/4 and 4/4 each waited ~12.4m. Therefore this issue remains open. Next work is consumer-side Docker fan-out reduction/changed-path routing, not reopening unsafe fleet width.

[rldyourmnd]

A second tenant's numbers, and they invert your conclusion — plus a measurement trap that bit me twice before I saw it.

Reporting from NDDev-it-com/setup-systems, which started using the fleet today, 2026-08-25 between 10:20Z and 11:45Z. Across 47 placed jobs in six ci waves:

pool	placed	median queue	max queue
nddev-linux-standard	31	2.5 min	19.1 min
nddev-linux-integration (Docker-capable)	16	4.7 min	8.3 min

So on this tenant the Docker-capable class has the higher median but the shorter tail, and the long waits are on the general-purpose pool. Same fleet, same hours, opposite concentration from the one your wave shows. Whatever distributes the work, it is not simply starving the Docker class.

The trap, which matters more than the numbers

Run 32842531323: pins and render were created at 11:28:59, never received a runner — steps: [], runner_name empty — and sat for thirteen minutes until the run was cancelled. GitHub reports that job as

created_at = 2026-08-25T11:28:59Z
started_at = 2026-08-25T11:28:59Z     ← identical
completed_at = 2026-08-25T11:42:09Z

started_at is set equal to created_at for a job cancelled before placement. So started_at - created_at scores it zero queue and thirteen minutes of work — precisely inverted. The job that waited longest and never ran looks like the healthiest one in the sample.

I read it wrong twice on the way here: first as a fourteen-minute queue from the status: queued field, then as thirteen minutes of work from the timestamps, before checking steps and runner_name and finding it had never been placed at all.

Your report already rejects one wrong denominator — sampling "time since the last placement". This is a second one hiding in the accepted formula, and it biases in the worst direction: the more the fleet fails to place a job, the better that job's queue metric looks, right up until someone cancels the run.

Suggested guard for whatever measures this next

Count a job as placed only when runner_name is non-empty or steps is non-empty. Everything else is unplaced time, and it belongs in its own column rather than in either queue or work. Otherwise a fleet that stops dispatching entirely reports a perfect queue — which is the same failure #146 is about detecting.

Not asking for anything on our side: our jobs are tiny, the waits cost us minutes, and nothing timed out that we did not cancel ourselves. Filing because the shape and the metric are worth having, and because we are a new tenant on this fleet and our numbers disagree with the existing ones.

[rldyourmnd]

Consumer Docker fan-out reduction is now merged as NDDev-it-com/almaty-libraries#954 after rebasing onto the release main and completing a second fully green quality, CodeQL and image-security wave.

Structural result remains: quality-gates integration jobs 8 -> 6; staging symbolication, gitleaks and the image-plan feeder no longer consume Docker capacity; the natural Docker Hub token 500 now receives three logged attempts.

Complete-wave evidence remains honest and does not close this issue:

run	integration jobs	total queue	total work	queue share
baseline 32823562365	8	4,588s	4,566s	50.1%
first green changed head 32838099991	6	20,774s	4,320s	82.8%
rebased green head 32842482273	6	12,184s	4,410s	73.4%

The second sample improved substantially over the first burst but remains slower than baseline because both standard and integration waited under estate-wide saturation. The separate My-Attention starvation defect is now resolved: GARM .80 skips blocked exact intents, the stale organization integration image was reconciled b5 -> b8, and nine natural backlog jobs completed on nine unique runners (github-device-sync-estate#230, closed).

Next evidence should be a natural post-.80 Almaty main/PR wave under the corrected estate. No unsafe overcommit or synthetic load is justified by these samples.

[rldyourmnd]

Natural post-.80 Almaty main wave completed green: run 32873536092, head d643a9de65e4f14be64d9e0a9bddc4cba23875df, 28/28 jobs terminal success. Correct placement guard (runner_name non-empty or steps non-empty) classified all 28 as placed; no cancelled-before-placement job was mis-scored.

Using the issue's existing run-created → job-started convention for direct comparability:

class	jobs	queue sum	work sum	queue share	median queue	max queue
all	28	93,711s	4,605s	95.3%	4,271s	5,300s
nddev-priority-standard	22	87,225s	3,138s	96.5%	4,392s	5,300s
nddev-priority-integration	6	6,486s	1,467s	81.6%	621s	2,277s

Interpretation: .80 dispatch is live and eventually placed every job; the scheduler-recovery controller reported healthy/no-stale-undispatched-instance every 30 seconds throughout. The binding path in this burst is now priority standard, not Docker integration. Four fleet hosts were under real CPU pressure while memory remained non-OOM; no failed systemd units were present.

Important limitation: this convention includes workflow DAG dependency wait because GitHub's jobs API exposes started_at but not an eligibility timestamp. It is comparable with prior numbers but is not pure runner-queue latency. The issue remains open: the natural sample is materially worse, and neither unsafe host overcommit nor cancelling durable main verdicts is acceptable.

[rldyourmnd]

Consumer-side standard-lane fan-out reduction is now merged as NDDev-it-com/almaty-libraries#960 (eedc3b980f9c97bbda9d0a0c7a3c050af55b6a9a). Admin Vitest changed 3→2 shards and infrastructure contracts 6→4; exact partition proof still covers all 1,116 discovered tests and every aggregate gate remains fail-closed. A live 3-shard attempt exceeded the 15-minute job bound, so it was rejected and corrected before merge.

The first natural main wave after merge, run 32890112794, completed 25/25 success. Comparable placed-job measurement:

metric	pre-change .80 run 32873536092	post-change run 32890112794	delta
jobs	28	25	-10.7%
total queue	93,711s	16,066s	-82.9%
total work	4,605s	4,121s	-10.5%
median queue	4,271s	685s	-84.0%
max queue	5,300s	1,513s	-71.5%
priority-standard queue	87,225s	14,699s	-83.1%
priority-integration queue	6,486s	1,367s	-78.9%

The natural result validates the bounded-fan-out change and shows the dispatcher/capacity path progressing normally. The issue remains open because standard-lane queue share is still 85.9% under this run-created-to-start convention, and that convention still includes DAG dependency wait; a pure eligibility-to-placement measurement is not yet available from GitHub jobs API.

[rldyourmnd]

Pure eligibility measurement is now implemented and deployed. Public PR #195 defines eligibility-to-start-by-job from durable pre-start phases (queued, assigned, acquiring, acquired) and excludes queue.running, so workflow DAG wait and execution time cannot enter the value. Estate rollout #248 deployed observer SHA 1d01c728… and a fourth versioned OpenObserve panel. Its natural integration job produced the first enriched exact row: run 32924250712, estate-integration, nddev-linux-integration, eligibility→start 50.336031s; query time 32ms.

The historical Almaty run 32890112794 predates workflow-run/job-name enrichment and cannot be reconstructed under the new exact grouping, so this issue remains open for the first natural post-cutover Almaty wave. No traffic or rerun was manufactured.

[rldyourmnd]

First natural Almaty activity after telemetry cutover is scheduled quality run 32927579444 (success). The schedule intentionally executed only two placed jobs; all other matrix/application jobs were skipped, so it is not the representative full wave needed to close this issue. Exact eligibility→start values were 50.548811s and 50.327056s on nddev-priority-standard (sum 100.875867s, query 31ms). This validates the measurement on the priority tenant but leaves the full main/PR cohort pending.