Gossip persistence - #65
Draft
plotnick wants to merge 8 commits into
Draft
Conversation
A sled that joins an established universe receives its entire history, and until now executed every retained job as its causal chain became ready. The gossip manager now publishes the causal frontier of the set received at join alongside the rumors handle, and the state machine executes only jobs that arrive as strict causal descendants of that frontier. Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>
A sled that reboots mid-job leaves the rack believing the job runs forever: the executor died with the process, so no stop event ever comes. Now, when replayed history at a universe join shows jobs running on our own baseboard that this incarnation is not actually running, we gossip an error event for each with the new ProcessError::Interrupted, closing them out honestly for every seat in the rack (sush#9). The scan runs only at local quiescence, after draining every message already delivered, so a job whose terminal event is present is never falsely interrupted. A stop that is still in flight elsewhere in the rack can race the declaration; the audit trail then carries both, and honestly. Jobs this incarnation really is running when its universe migrates are excluded: they continue, and their events land in the new universe. Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>
A rumors bookmark persists a peer's identity so a restarted sled reclaims it instead of stranding it. We owe rumors raw byte storage with two guarantees: stores commit atomically, and a load never returns a record older than the newest store we acknowledged, since a stale record claims coverage the identity already transmitted past and corrupts causality on reclamation, whereas a lost record merely strands. Records live in a small sequence-numbered envelope, one file per boot M.2 slot. Loads read every slot and the highest sequence wins, tolerating boot-device flips and disk swaps; stores go only to the winning slot, so one disk's health is in the write path rather than two. A BookmarkSource hands out one handle per peer: minting a handle supersedes all earlier ones, so a straggling store from an abandoned universe cannot clobber its successor's record. Shed handles persist nothing, for peers that must keep gossiping after their storage failed, and the null source persists nothing at all, for the standalone server and tests. Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>
Every gossip peer now carries a bookmark handle. The seed's is attached at creation, where a pristine peer costs nothing; a migration mints a fresh handle for the joined peer, superseding the abandoned universe's. Identities recorded by previous incarnations are reclaimed at the first gossip after a migration returns us to the universe that knew them. Persistence must never cost availability. A bookmark failure inside a session aborts that session before any wire traffic, so a seed on bad storage could never even learn it lost dominance: probe the storage at seeding and shed the bookmark up front. A join whose identity cannot be persisted likewise sheds and gossips on. Both degradations merely strand identities, which is what every restart did before this commit. A store that fails after a successful join still stops that sled's gossip until its next migration or restart; the planned rumors API for shedding a live peer's bookmark is the remaining fix. Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>
The adversarial review found the orphan scan drawing candidates from every job running on our baseboard, so a replayed straggler arriving while a live job ran would declare that live job interrupted, break its attachment, and discard its eventual result. Candidates now come only from start events that themselves arrived as replayed history: a job this incarnation started can never be a candidate, no matter when the scan runs. Three more review findings ride along. A genuine stop and an interrupted declaration now converge to the real result in either arrival order (errors no longer displace terminal statuses, and a stop supersedes an interrupted error), instead of the interrupt winning both races. A job we declared interrupted no longer counts as a survivor at the next universe swap, so back-to-back migrations re-declare it rather than exempting it forever. And replayed messages no longer mint local cancelled statuses or re-gossip concurrent-session errors, which grew the set a little more on every rejoin by every sled. Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>
The adversarial review found the one write path the atomic rename does not serialize: a store future dropped at its await (a migrating manager aborts session drivers; a join can time out) detaches the blocking write, whose rename then lands on top of a newer record that already returned Ok. Renames now go through a commit fence: a write lands only if its sequence number exceeds everything this process has committed, so a straggler loses instead of clobbering. The envelope also gains a digest over the sequence number and record, since the whole safety argument rides on an integer that was previously trusted straight off the disk. Loads no longer regress the in-memory sequence reservation or serve superseded handles, the probe proves a slot writable by writing instead of guessing from directory metadata, and a bookmark failure inside a gossip session now warns loudly: it stops every later session at the persist gate, so unlike routine link churn it must not hide at debug level. Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>
A job whose process died with a previous incarnation but whose bookkeeping says it still runs is exactly what Unix calls a zombie, and recording its death is reaping. Rename replayed_started, orphaned_jobs, and interrupt_orphans accordingly, and sweep the recently added comments for style: no more spliced clauses, one home per argument, and the supersede vocabulary instead of fences. Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>
The ipcc feature gate merged upstream as 98615b86, identical in content to the branch rev we carried; the pin now names the commit on sprockets main. Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Just identity bookmarks so far, but full rumors set persistence coming soon!