Fix #113: a dead wire says so, the page re-dials on evidence, and subduction survives the rebind - #122
Merged
Conversation
conn_results was write-once: the handshake outcome went in (wire error,
keyhive bad-peer, or Ok(peer)) and nothing ever observed the connection
dying afterwards -- iroh_writer/iroh_reader break out of their loops
silently on transport errors, and nobody called the iroh WIT's
wait-closed. A dead wire was indistinguishable from a healthy one, so
no page path could ever re-dial without risking the double-dial the
direction discipline forbids.
Now every connection that gets past wire setup has a fourth task:
conn_gone_monitor awaits conn.wait_closed() and writes Err('gone: ...')
into conn_results -- overwriting ONLY an Ok entry (a wire/handshake
error is more informative than 'gone' and survives; a missing entry
belongs to a handshake still settling). The gone message carries
close-info when the peer sent one; observed in practice it never does
('gone: transport closed (no close frame)') -- a peer whose relay died
sends nothing.
engine.wit's conn-status doc comment now spells the contract: Ok(Some)
= handshake ok and last known alive; Err starting with 'gone:' = the
wire came up and later died (latched; a re-dial mints a new conn id);
other Err = the handshake itself failed. The gone: prefix is a
machine-readable marker, not phrasing.
Measured (demo/host/conn-gone-check.ts, new committed gate + justfile
recipe): relay SIGKILL -> gone on both sides in ~0.1s, three runs. The
dispatch's idle-timeout theory was wrong for this case: the relay leg
is a websocket over TCP, so killing the relay closes the socket and the
endpoint learns synchronously; the QUIC idle timeout governs only the
no-socket-death shapes (blackhole, cut path). Both mechanisms belong to
the pinned endpoint; the probe's bound (20s) deliberately covers the
slow one.
What a TS caller sees through runtime/engine.ts: connStatus THROWS
'component error: gone: ...' -- match with includes('gone:'), not
startsWith.
Gates: engine just check (clippy -D warnings) clean; just compose;
conn-gone x3 green; solo-pairing device-pairing solo-resume-sync 3/3
(ceremony waits unregressed); relay-partition still red in the same
final act (the page half is what re-dials -- next commit).
… into fix/113-conn-gone
…ence The solo page acts on the engine's gone marker. The resumed-boot-only resumeWire becomes the page's ONE standing wire-keeper, entered from three places -- resumed boots, and both ceremonies' finally (a ceremony that fails mid-flight is exactly the page that most needs the loop; the success-only arming was a bug caught in development). One tick: health-sweep the dial handle and the acceptor (one enqueue slot each, ms-sized -- #115's discipline), where 'gone:' -- and ONLY 'gone:' -- clears that wire's state so the next tick re-dials or reposts; a non-gone error goes to onError; settling is left alone. A missed death costs one tick; an invented one costs #78's double-dial, which is why gone is the single trigger and why the rebind fires only on the Closed evidence a repair attempt itself raises (a rebind takes every live connection with it -- never on suspicion). Subscriptions die with their connection: forgetting a wire also forgets its tasksWired entries, or the transport heals and zero bytes move (hit in development, kept as a comment). Measured along the way: a relay death latches the device's own endpoint Closed, so the re-dial path runs rebindEndpoint first (same address; the key pair is persisted). A vanished peer (its page reloaded) is detected in ~35s -- ~30s QUIC idle timeout + one 5s tick -- and re-dialled in under a second. one-sided-reload: PROMOTED to green (was expected: red since wave 1). The reader now learns its handle is stale and re-dials; heal measured at 35s, three consecutive runs 37.8-38.1s. Banner rewritten as the positive claim with the red history kept.
…rebind, and the relay-partition pins go green The wire-keeper moved #113's residue into the engine: after a relay bounce both sides rebound, re-dialled, and reported LIVE connections -- and the reader's sync-start still never settled. Four links, all read in the pinned subduction source, none assumed: 1. QueueTransport held its own channel ends (lib.rs), so when iroh_reader/iroh_writer died with the wire, recv_bytes parked forever and send_bytes succeeded into a void -- a transport that CANNOT fail. 2. Subduction's teardown is driven entirely by transport failure (connection_loop's exit is the only path to remove_connection), so a transport that cannot fail is never removed. 3. add_connection APPENDS to the peer's connection list -- dead at index 0, live at index 1. 4. sync_with_peer walks that list serially under this engine's own NeverTimeout: the call on the corpse parks forever and the live connection is never reached. Only the CALLER of sync_with_peer walks the list -- an acceptor replies on the connection the request arrived on -- which is the whole dialler/acceptor asymmetry, and why one-sided-reload was green while the relay pins stayed red. The fix tells the queues the truth: conn_gone_monitor now closes a dead connection's inbound senders (subduction + keyhive wires, tracked in new conn_inbound state) BEFORE the status write -- async-channel close drains queued frames first, so nothing is lost -- and subduction's own machinery does the rest. Also closed, from review: the monitor race where a conn dying between handshake success and the outcome write was latched alive forever (the monitor now waits for the entry; the outcome write consults conn.state() as belt and braces). demo/host/rebind-sync-check.ts (+ just rebind-sync) is the headless gate that pinned it: red before (sync never settled on either side, 30s x forever), green after (settle 0.20s, todos cross both ways in 0.03-0.06s, three runs). relay-partition and relay-partition-asym: PROMOTED to green after three waves of red, banners carrying the whole history (wave 1: no loop, no signal -- #113 filed; wave 2: the page wire-keeper, which moved the gap; wave 3: this chain). Engines converge 4.3-5.4s after the relay returns; the rendered-rows read is its own act now, bounded -- the engines agreeing and the screens following are different claims. Five consecutive green runs of both. Left deliberately alone, flagged for its own issue: NeverTimeout, ours not upstream, turns every slow peer into wedged-forever; a real bound deserves its own measured decision. Gates: engine just check (clippy -D warnings) clean; conn-gone and rebind-sync green x3; full e2e 34/34 (no xfails left); devstore matrix ALL REQUIRED ROWS PASS; soak seed 2/25 green. Two independent review rounds; all should-fixes applied.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #113 — all three expected-red scenarios are now green and promoted; the suite has no xfails left.
Three commits, three layers:
Engine: the gone marker (dc28186). Every connection past wire setup gets a monitor awaiting the iroh WIT's latched
wait-closed;conn_resultsflips toErr("gone: …")— only ever clobbering anOk(a handshake error is more informative and survives).conn-status's contract is documented in engine.wit: thegone:prefix is machine-readable. New headless gatejust conn-gone: relay SIGKILL → gone on both sides in ~0.1s (the relay leg is a TCP websocket — synchronous; the QUIC idle timeout governs only no-socket-death shapes), latched.Page: one wire-keeper (bad7ba9).
resumeWiregeneralizes into the page's single standing keeper, entered from resumed boots and both ceremonies'finally.gone:is the only re-dial trigger (a missed death costs one 5s tick; an invented one costs #78's double-dial); rebind only on theClosedevidence a repair attempt itself raises; subscriptions die with their connection. Measured: peer-vanish detected ~35s (QUIC idle timeout + tick), relay-bounce wire live ~10s after the relay returns. one-sided-reload promoted to green (heal 37.8–38.1s ×3).Engine again: the stale transport that could not fail (e66850d). The keeper exposed the residue:
QueueTransportheld its own channel ends so it could never fail; subduction's teardown fires only on transport failure;add_connectionappends;sync_with_peerwalks the list serially under our ownNeverTimeout— parking forever on the corpse. Only the caller walks the list, which is the whole dialler/acceptor asymmetry. Fix: the gone monitor closes a dead connection's inbound queues (async-channel close drains first — nothing lost), and subduction's own removal machinery does the rest. Plus the review-caught race (conn dying between handshake success and the outcome write was latched alive forever) closed two ways. New headless gatejust rebind-sync: red before (sync never settles either side), green after (settle 0.20s, cross 0.03–0.06s). relay-partition and relay-partition-asym promoted to green after three waves of red — engines converge 4.3–5.4s after the relay returns; banners keep the full history.Flagged for follow-up, deliberately untouched:
NeverTimeout(ours, not upstream) turns every slow peer into wedged-forever; a real bound deserves its own measured decision — issue to follow.Gates: engine clippy
-D warningsclean;conn-gone+rebind-sync×3; full e2e 34/34; devstore matrix ALL REQUIRED ROWS PASS; soak seed 2/25 green. Two independent review rounds; all should-fixes applied.Automerge armed (merge commit, org convention).