Repository navigation
Conversation
…ale quarantine Observed full-catalog pace is 6-14 h (disk/versions/shards cases run 5-30 min each), past the 540-min loop budget — raise the loop to 900 and the job to 990 min. The runner service is occasionally stopped by host maintenance mid-case (journal: stops at 06:52, 07:18-11:19, 12:32 on Nov 4 each killed an in-flight sweep). A killed run leaves its quarantine markers and blocks every later admission until someone recovers manually. Admission now checks marker ownership: markers whose run is no longer queued/in_progress are recovered (stop rustfs, purge, remove marker) and admission retries once; live quarantine is never touched.
Collaborator
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two operational fixes from today's three full-run attempts:
1. Time budget. Observed full-catalog pace is 6–14 h (admin cases 2–5 min; disk/versions/shards families 5–30 min). The 540-min loop budget would truncate the catalog even on a clean run. Loop → 900 min, job → 990 min.
2. Self-recover stale quarantine. The runner service was stopped by host maintenance three times today (journal: 06:52, 07:18–11:19, 12:32), each kill leaving quarantine markers + a live rustfs deployment that blocked every later admission (the 07:06 scheduled run died exactly there). Admission now inspects marker ownership on failure: markers whose owning run is no longer queued/in_progress are recovered (stop rustfs, purge, remove marker) and admission retries once; markers owned by a live run are never touched.