Problem
An ephemeral worker was deregistered while it still owned a running job. The job did not fail, did not cancel, and did not time out — it stayed in_progress indefinitely, holding a required check pending and blocking a merge with no signal that anything was wrong.
timeout-minutes does not rescue this. GitHub enforces that limit through the runner agent, so when the agent is gone there is nothing left to enforce it. The job outlived its own declared timeout and kept running.
No private repository, host, address or credential is required to describe the failure.
Measured
Observed 2026-08-26 on an nddev-priority-integration worker:
| fact |
value |
| runner name |
nddev-eqoccpmu3aev |
| runner id |
23061 |
GET /orgs/{org}/actions/runners/23061 |
404 — not registered |
| job status |
in_progress, conclusion: null |
| job started |
2026-08-26T13:36:00Z |
| still running after |
50+ minutes |
declared timeout-minutes |
45 |
| step reached |
build step, never advanced |
The worker is gone from the organization's runner list while the job it was executing is still, according to the API, executing.
This is not a capacity or a class problem
The same job shape ran 65 times successfully on the same day, longest 4.9 minutes. Three earlier runs of this exact job name completed in ~2.5 minutes each. One worker vanished; the rest of the class was healthy throughout. So this is a lifecycle defect, not saturation and not a bad image.
Why it matters more than one lost job
The job is unrecoverable from the consumer side without discarding work:
- a required check stays
pending forever, so the pull request can never merge;
- ordinary
gh run cancel does not clear it;
- the only recovery is
force-cancel plus a full re-run, because a partial --failed re-run breaks any workflow whose artifacts are keyed by run_attempt — the preserved successes upload under attempt 1 while the re-queued aggregator looks for attempt 2.
So one vanished worker costs a full re-scan of every image in the wave, not one job.
Required behavior
- when a worker is deregistered or its instance is destroyed, fail or cancel any job it still owns rather than leaving it
in_progress;
- treat "job assigned to a runner that no longer exists" as a reconcilable state and reconcile it on a bounded interval;
- surface it — a job whose runner has vanished should be visible as an incident, not as a job that is merely slow.
The distinction that makes this detectable: runner liveness is not job liveness, the same shape as #146's "process health is not scheduler progress".
Problem
An ephemeral worker was deregistered while it still owned a running job. The job did not fail, did not cancel, and did not time out — it stayed
in_progressindefinitely, holding a required check pending and blocking a merge with no signal that anything was wrong.timeout-minutesdoes not rescue this. GitHub enforces that limit through the runner agent, so when the agent is gone there is nothing left to enforce it. The job outlived its own declared timeout and kept running.No private repository, host, address or credential is required to describe the failure.
Measured
Observed 2026-08-26 on an
nddev-priority-integrationworker:nddev-eqoccpmu3aev23061GET /orgs/{org}/actions/runners/23061in_progress,conclusion: null2026-08-26T13:36:00Ztimeout-minutesThe worker is gone from the organization's runner list while the job it was executing is still, according to the API, executing.
This is not a capacity or a class problem
The same job shape ran 65 times successfully on the same day, longest 4.9 minutes. Three earlier runs of this exact job name completed in ~2.5 minutes each. One worker vanished; the rest of the class was healthy throughout. So this is a lifecycle defect, not saturation and not a bad image.
Why it matters more than one lost job
The job is unrecoverable from the consumer side without discarding work:
pendingforever, so the pull request can never merge;gh run canceldoes not clear it;force-cancelplus a full re-run, because a partial--failedre-run breaks any workflow whose artifacts are keyed byrun_attempt— the preserved successes upload under attempt 1 while the re-queued aggregator looks for attempt 2.So one vanished worker costs a full re-scan of every image in the wave, not one job.
Required behavior
in_progress;The distinction that makes this detectable: runner liveness is not job liveness, the same shape as #146's "process health is not scheduler progress".