Problem
A production control plane remained active/running and continued serving metadata and callbacks, but newly persisted pending_create instances stopped reaching provider create. Restarting only the manager service immediately resumed dispatch. This is a portable scheduler liveness defect: process health is not scheduler progress.
No private repository, host, address or credential is required to describe the failure.
Required behavior
- persist scheduler progress/heartbeat independently from HTTP process health;
- detect queued create work whose dispatch age exceeds a bounded threshold while provider capacity is eligible;
- restart or reconstruct the affected worker loop idempotently without restarting unrelated services;
- never create a duplicate active worker during recovery;
- expose unhealthy, recovering and recovered states in metrics/logs;
- preserve the original intent and attempt history;
- add deterministic restart/replay tests and a natural-job canary.
Acceptance
A fault-injected stopped dispatcher recovers without operator restart, every accepted intent reaches one terminal outcome, and the test proves zero duplicate provider create or double release.
Problem
A production control plane remained
active/runningand continued serving metadata and callbacks, but newly persistedpending_createinstances stopped reaching provider create. Restarting only the manager service immediately resumed dispatch. This is a portable scheduler liveness defect: process health is not scheduler progress.No private repository, host, address or credential is required to describe the failure.
Required behavior
Acceptance
A fault-injected stopped dispatcher recovers without operator restart, every accepted intent reaches one terminal outcome, and the test proves zero duplicate provider create or double release.