One job intent on nddev-linux-fast has been assigned for 101 minutes and never served, while the same class served at least fourteen intents assigned after it - six of them assigned 66 minutes later and served within five minutes. The intent is still alive and still reconciling on its cycle; it is not lost, not terminal, and not backed off. It is skipped.
This is not #197. That one was a stalled create path with a leaked retry record and it self-cleared; this is a single intent passed over while its class works.
No private repository, host, tenant or credential is required to describe it. Two accounts share the queue below and are called A and B.
The ordering
Assign time against start time, same class, taken from the manager journal:
| intent |
account |
assigned |
started |
waited |
1f0dbfbe |
B |
11:02:23Z |
11:04:54Z |
2m31s |
f7b22c5d |
A |
11:12:09Z |
11:16:22Z |
4m13s |
161c5248 |
B |
11:12:24Z |
12:14:27Z |
62m03s |
782eab20 |
A |
11:17:10Z |
11:19:46Z |
2m36s |
342b3571 |
B |
11:20:58Z |
never |
101m and counting |
84159fa5 |
A |
11:28:13Z |
12:20:10Z |
51m57s |
81a3be77 |
A |
11:28:14Z |
12:21:02Z |
52m48s |
86718ff7 |
A |
11:28:14Z |
12:21:10Z |
52m56s |
65a3dea0 |
A |
11:28:14Z |
12:22:15Z |
54m01s |
ca50745a |
A |
11:33:53Z |
12:22:32Z |
48m39s |
82a9a46a |
A |
11:33:54Z |
12:22:48Z |
48m54s |
1ff96504 |
A |
11:33:54Z |
12:27:48Z |
53m54s |
e6ddbd23 |
A |
12:25:11Z |
12:30:05Z |
4m54s |
232151a4 |
A |
12:27:24Z |
12:31:47Z |
4m23s |
bb85563a |
A |
12:27:24Z |
12:31:47Z |
4m23s |
73471dad |
A |
12:27:34Z |
12:32:15Z |
4m41s |
54b17bf4 |
A |
12:27:34Z |
12:33:17Z |
5m43s |
7b2e2b9a |
A |
12:27:34Z |
12:35:39Z |
8m05s |
7cbd91f1 |
A |
12:27:36Z |
12:35:39Z |
8m03s |
The skipped intent is not the oldest and not the newest. Its own sibling from the same account, assigned 8 minutes earlier, was served at 12:14:27Z as the first job of that burst - so the account is not blocked either.
It is alive the whole time
It has been re-emitting reconciled authoritative queued job intent on its cycle throughout, most recently 12:55:58Z, 95 minutes after assignment. There is no retry record for the class at all right now, and the pool holds zero leases, so nothing is holding a slot on its behalf and nothing has classified it as failing. Every observable says it is a healthy queued intent that the dispatcher does not pick.
The class serves in bursts, which is what hides it
Creates and starts on this class per 10-minute bucket over the preceding six hours - every other bucket in that window is empty:
10:5x 9 12:1x 1
11:0x 2 12:2x 12
11:1x 3 12:3x 13
26 starts in six hours against 142 on another class in the same window, in two bursts of about 25 minutes each, with nothing since 12:37:08Z. Both matchers - the scale-set worker name and the class name on the line - agree exactly on those counts, so this is not a measurement artefact.
A skipped intent is invisible against that: each burst looks like a queue draining, and the one intent that never drains reads as "still waiting its turn" until you sort by assign time.
One correction to #197
The job named in that issue's parked retry record, 478cce5a, was assigned 11:12:09Z and started 11:16:43Z. It ran. The record outlived a job that had already succeeded, which settles the "leak rather than backoff" reading there: a terminal outcome did not clear its retry entry.
What would catch it
The queue-age predicate in #146 fires here, but only after the intent is old - and on a bursty class that is hard to tell from an ordinary busy period. The narrower one is an ordering invariant, checkable at every dispatch:
no intent may be served while an intent for the same class, admitted earlier and still queued, is skipped more than N times
Here that is violated fourteen times and would have fired at 12:20:10Z, the first out-of-order start, forty minutes before this report.
Not requested
Nothing was changed, no unit restarted, no recovery command run. The intent is still queued as of 13:01:59Z, so the state is inspectable right now if that is useful.
What it costs us: the skipped intent is a required merge context. Two pull requests are held behind it with auto-merge armed, and we are not lifting branch protection.
One job intent on
nddev-linux-fasthas been assigned for 101 minutes and never served, while the same class served at least fourteen intents assigned after it - six of them assigned 66 minutes later and served within five minutes. The intent is still alive and still reconciling on its cycle; it is not lost, not terminal, and not backed off. It is skipped.This is not #197. That one was a stalled create path with a leaked retry record and it self-cleared; this is a single intent passed over while its class works.
No private repository, host, tenant or credential is required to describe it. Two accounts share the queue below and are called A and B.
The ordering
Assign time against start time, same class, taken from the manager journal:
1f0dbfbef7b22c5d161c5248782eab20342b357184159fa581a3be7786718ff765a3dea0ca50745a82a9a46a1ff96504e6ddbd23232151a4bb85563a73471dad54b17bf47b2e2b9a7cbd91f1The skipped intent is not the oldest and not the newest. Its own sibling from the same account, assigned 8 minutes earlier, was served at 12:14:27Z as the first job of that burst - so the account is not blocked either.
It is alive the whole time
It has been re-emitting
reconciled authoritative queued job intenton its cycle throughout, most recently 12:55:58Z, 95 minutes after assignment. There is no retry record for the class at all right now, and the pool holds zero leases, so nothing is holding a slot on its behalf and nothing has classified it as failing. Every observable says it is a healthy queued intent that the dispatcher does not pick.The class serves in bursts, which is what hides it
Creates and starts on this class per 10-minute bucket over the preceding six hours - every other bucket in that window is empty:
26 starts in six hours against 142 on another class in the same window, in two bursts of about 25 minutes each, with nothing since 12:37:08Z. Both matchers - the scale-set worker name and the class name on the line - agree exactly on those counts, so this is not a measurement artefact.
A skipped intent is invisible against that: each burst looks like a queue draining, and the one intent that never drains reads as "still waiting its turn" until you sort by assign time.
One correction to #197
The job named in that issue's parked retry record,
478cce5a, was assigned 11:12:09Z and started 11:16:43Z. It ran. The record outlived a job that had already succeeded, which settles the "leak rather than backoff" reading there: a terminal outcome did not clear its retry entry.What would catch it
The queue-age predicate in #146 fires here, but only after the intent is old - and on a bursty class that is hard to tell from an ordinary busy period. The narrower one is an ordering invariant, checkable at every dispatch:
Here that is violated fourteen times and would have fired at 12:20:10Z, the first out-of-order start, forty minutes before this report.
Not requested
Nothing was changed, no unit restarted, no recovery command run. The intent is still queued as of 13:01:59Z, so the state is inspectable right now if that is useful.
What it costs us: the skipped intent is a required merge context. Two pull requests are held behind it with auto-merge armed, and we are not lifting branch protection.