tart_health_agent: collect the worker_exit_69 semaphore - #71
Merged
Conversation
Companion to the hangar change that replaces the guest-vs-tart-run uptime ratio as the crash-loop signal. worker-runner.sh writes /opt/worker/worker_exit_69 the first time generic-worker exits 69 (the "cannot free enough disk" path) and deletes it on any other exit, logging "Worker recovered from previous exit code 69". So its presence means the last worker exit was the panic path and its age means how long that has been true — which is exactly the crash loop, with no heuristic involved. Emitted only when the file exists, so an absent semaphore arrives as None rather than 0; 0 would otherwise read as "just started looping". Verified end to end against macmini-m4-236 with --dry-run: slot 1 mac-c51932 worker_exit_69_age_s = None (healthy, no semaphore) slot 2 mac-962a94 worker_exit_69_age_s = 2566 (semaphore backdated 42m) and the test semaphore was removed afterwards. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Companion to the hangar PR that replaces the guest-vs-tart-run uptime ratio as the crash-loop signal.
worker-runner.shwrites/opt/worker/worker_exit_69the first time generic-worker exits 69 (the"cannot free enough disk" path) and deletes it on any other exit, logging "Worker recovered from
previous exit code 69". So:
That is the crash loop directly, with no heuristic.
worker-runner.shitself usesfind -mmin +15on the same file to mean "problem repeated for 15m", so hangar reuses that boundary.
Emitted only when the file exists, so an absent semaphore arrives as
Nonerather than0—0would otherwise read as "just started looping".Verified end to end
--dry-runagainst macmini-m4-236 from a staged copy (deployed tree untouched, confirmed 0 modifiedfiles afterwards):
worker_exit_69_age_smac-c51932Nonemac-962a942566The test semaphore was removed afterwards and its absence confirmed.
🤖 Generated with Claude Code