Repository navigation
A watchdog for the one question no instrument asked - #4495
Merged
Merged
Conversation
The agent server crash-looped, spent Railway's restartPolicyMaxRetries budget,
and the platform stopped restarting it. `/queen/status` then answered
`{"status":"error","code":502,"message":"Application failed to respond"}` for
NINE HOURS with the whole swarm stopped.
Every instrument built this week reads the swarm's OWN numbers. The pusher
exits 2 when the endpoint is silent - one red scheduled run, nothing else.
Nothing asked whether it answers at all.
watchdog.py asks every five minutes and is the only thing here allowed to
restart the service. It refuses on one bad probe, because a deploy answers 502
too; refuses without a token, rather than pretending to have healed anything;
and refuses to ask twice in a row, because a redeploy that did not help is not
fixed by another. The alarm issue closes itself when the swarm answers.
The probe is tested against the platform's own 502 body, which is valid JSON
and is not a swarm: a 200 carrying `workers` is the only yes.
CENSUS: quiet 60 -> 61 workflow files; shell 60 -> 61 files, 81 -> 82 jobs,
273 -> 275 run-steps. Classified in check_pr_branch_filters.py in this commit.
Closes #4494
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
gHashTag
enabled auto-merge (squash)
September 21, 2026 04:56
Contributor
Contributor
|
📓 NotebookLM Notebook linked to this PR
This notebook contains session context, decisions, and artifacts for this work. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #4494
Nine hours of that, with the whole swarm stopped. The agent server
crash-looped, spent Railway's
restartPolicyMaxRetries: 10— a budget, not apromise — and the platform stopped restarting it.
Every instrument built this week reads the swarm's own numbers. The pusher
exits 2 when the endpoint is silent: one red scheduled run, nothing else.
Nothing asked whether it answers at all.
What it refuses
The alarm issue closes itself when the swarm answers again.
One of those three is the platform's own 502 body: valid JSON, and not a swarm.
A 200 carrying
workersis the only yes.Both ledgers a new workflow moves are updated in this same commit.
To have it restart the service by itself, three secrets are needed —
RAILWAY_TOKEN,RAILWAY_SERVICE_ID,RAILWAY_ENVIRONMENT_ID.🤖 Generated with Claude Code