Repository navigation
chore(release): v0.30.0+ciris.1 — direct links through NAT (#70), link telemetry (#77) - #78
Conversation
…ies on CIRISEdge clears its per-link attribution at LinkClosed (CIRISEdge#810). It therefore relies on this: a receiver-side resource whose link closes while its off-lock assembly (#71) is running fails exactly once, with ResourceFailed{LinkClosed}, and that failure is emitted before LinkClosed. The assembly's late result then yields no ResourceCompleted and no second failure. The #71 test only checked that some failure appeared and that the late result delivered nothing. It now asserts the full invariant. A variant covers a close sent by the peer. Mutation-checked: when remove_link stops taking the assembling resource, both variants fail. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BXj7C4ZeFuczxxDVadAHZ5
…e#819, leviculum#57) A field gauge for link accumulation. NodeCore::link_census, and ReticulumNode::link_census in the driver, return the established links (Active or Stale) split into initiator and responder, a count of pending ones, and the same split per destination, most links first. It names the owner of a growing count. The initiator keeps a link alive with its keepalives, so a rising initiator count toward one destination means this node is not closing what it dialled. A rising responder count under its own destination means the peers are not closing theirs. CIRISEdge#819 traced a canonical's 1024 links per 35 min to the first case: its dial pool grows to demand and never shrinks. Before that, diagnosing it took a reading of the reap rules from the source. One walk of the link table under a brief core lock. Added to the lock census, together with known_destination_evictions (leviculum#49), which should have been added when that method landed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BXj7C4ZeFuczxxDVadAHZ5
… facade (leviculum#77) The P0 and P1 leviculum asks of CIRISServer's unified-telemetry FSD (PR #746, §4). The maintainer has made instrumentation the priority before the next scale step. P0 - ReticulumNode::link_list (NodeCore::link_list): every link with its caller-visible id, destination, role, state, age, idle seconds, RTT and interface. - Cumulative lifecycle counters, link_lifecycle: established by role, closed by LinkCloseReason, and handshake_failed by reason for links that never established. remove_link notes whether the departing link had established, and emit_link_closed counts it. Every close follows its removal, so established - closed equals the live census. - Divergence alarm. Every 10 s the event loop compares the completion mirror with the core's established count (MirrorWatch). A disagreement that outlasts 30 s raises one LINK_MIRROR_DIVERGED event (catalogued) per episode, counted in link_count_check().alarms. - The per-destination census from the previous commit. P1, facade per FSD §5: `metrics` in the library, the exporter in the host - The `metrics` crate (default-features off; its one dependency is rapidhash). ReticulumNode::publish_metrics records the lifecycle, the census, the mirror check, TransportStats (drops by every DropReason, through the new TransportStats::drops_for), PlaneStats, the known- destination evictions and the core's heap census. That last one gives std hosts per-component memory and a mean bytes-per-link. It is pull based: the host calls it on its scrape cadence. Counters use absolute(), so a scrape reads the node's own totals. - telemetry::METRIC_CATALOG lists 24 metrics, each with a kind, a unit, its allowed label keys and a description; describe_* is registered from it once. Every label value is a bounded enumeration. No link, destination or peer id is ever a label. - event_catalog_completeness now scans metric sites too. Every recorded name must be catalogued, every catalogued name recorded, and every label key allowed. The source walk is shared with the event scan. Housekeeping - THIRD-PARTY-NOTICES regenerated for metrics and rapidhash. This also picks up a stale rustls version line that was already wrong on main. - leviculum-nrf/Cargo.lock now carries the fork's version string, which the release bumps never updated. cargo-about --frozen failed on it. - The lock census adds the new methods (TOTAL 65). - event_log.rs's apply_inbound citation already pointed at unrelated code on main; it is corrected. - The doc-citation guard reports no drift beyond what main already has. Tests: mvr_link_census (list, counters by reason, handshake failure, and the counters agreeing with the census); drops_for over every reason; MirrorWatch (grace, one alarm per episode, re-arm); the metric scanner; and telemetry_publish, which runs two real nodes over TCP through the whole life of a link with a capturing recorder. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BXj7C4ZeFuczxxDVadAHZ5
…td (Codex on #76) LinkCensus, DestinationLinks, LinkInfo, LinkLifecycle, LinkRole, LINK_CLOSE_REASONS, link_close_reason_name, LinkState and NodeHeapCensus are now nameable from leviculum_std, as LinkStats and TransportStats already were. A consumer without its own leviculum-core dependency can use them in annotations and fields. telemetry_publish names every result through leviculum_std alone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BXj7C4ZeFuczxxDVadAHZ5
…is None, describes per publish, process-total counters
- reject_link counts the refused request, as LinkLifecycle::rejected and
as leviculum.link.handshake_failed{reason="rejected"}. remove_link
noted the departing link but reject_link reports no close, so refused
handshakes vanished from the metrics.
- LinkInfo::rtt_ms is Option<u64>: None until a sample exists, as in
LinkStats. Before, a pending link reported the 500 ms working fallback
as though it had been measured.
- Units and descriptions are registered on every publish instead of once
per process, so a recorder installed (or swapped for a local one) after
the first publish still learns them.
- Counters are process totals. Each node keeps the totals it last
published (CounterBaseline, held by ReticulumNode) and adds its growth
to one process-wide sum per series, which absolute() records. Two nodes
in one process now sum, and a replacement node carries on instead of
sitting under its predecessor's total, which absolute's keep-the-
maximum contract had caused. A recorder installed late still reads full
totals. Plain per-node increments would have lost exactly that.
Tests: the census test covers a refused request. telemetry_publish checks
that a late recorder receives every description, that two nodes sum into
one recorder, and that an established link has a measured RTT.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BXj7C4ZeFuczxxDVadAHZ5
…ng else is closed (Codex on #76) reject_link removed whatever link it was given, without a close. On an established or locally dialled link, that left the completion mirror and the lifecycle counters believing the link alive, and broke established - closed == live. Now: - a pending inbound request is refused and counted as rejected; - any other link is closed the way close_link closes it (close packet, LinkClosed with reason Normal, counted as a close). Its events go out with the node's next output. Test: rejecting a live link counts a close, not a refusal, and the counters still agree with the census. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BXj7C4ZeFuczxxDVadAHZ5
… rns-rs (leviculum#70)
Two nodes whose link runs through a transport node can now move that link
onto a direct UDP path through both NATs. Each side learns its reflexive
address from a facilitator, the pair swap addresses as channel signals on
the link they already share, and both punch at once. On success the link
moves onto a new point-to-point interface, and that interface announces
the node's destinations, so the path drops to one hop. On failure
(symmetric NAT, common on cellular) nothing changes. If the direct
interface dies later, the link goes back to the interface it had.
The wire protocol is rns-rs's DirectLink, implemented independently:
channel signals 0xFE00..=0xFE04 with the same msgpack bodies, RNSP and
STUN probes, the RNSH/RNSA punch frames, and the HKDF punch token. rns-rs
is under the Reticulum License, whose extra conditions AGPL code cannot
take on, so none of its source is copied. Its encoder supplied golden
vectors, which match byte for byte. Interop was verified against rns-rs
0.7.3 (98205ea) in all three combinations: rns-rs rnsd as relay and
facilitator, an rns-rs peer accepting a leviculum proposal, and an rns-rs
peer proposing to leviculum. Each time, data crossed the punched path
after the relay was killed.
- core: direct_link::{wire, session} (pure codecs and a sans-I/O state
machine) and node::direct_link (policy, signals taken off the channel
before the application sees them, a job queue for the driver, link
re-attachment and fallback). New NodeEvents DirectLinkEstablished,
DirectLinkFailed and DirectLinkLost.
- std: probe client, punch loop, a facilitator answering RNSP and STUN,
and interfaces::direct_udp. The direct interface is leaf-only, sends a
keepalive every 30 s, goes down after 120 s of silence, and answers a
late punch so a peer whose last ack was lost still completes.
- Off by default: policy `reject`, no facilitator. Config keys probe_port,
probe_addr, probe_protocol and direct_connect_policy, or
ReticulumNodeBuilder::direct_link. ReticulumNode::propose_direct_link
starts an upgrade. An upgrade is never automatic: a Python RNS peer
proves the signal and then stalls its channel on the unknown MSGTYPE.
Gate bookkeeping:
- scripts/ignored-counts.txt pins 3 ignored tests in direct_link_upgrade.
They are the rns-rs interop arms. No tier can run them, because each
needs an rns-rs process started beside it. They are run by name with
`-- --ignored` and the LEVICULUM_INTEROP_* variables their doc comments
give. The leviculum-only arms in that file run in every tier.
- scripts/core-lock-census.txt adds propose_direct_link and
direct_link_interface, plus known_destination_evictions (leviculum#49),
which should have been added when that method landed.
- Fork additions sit at the ends of upstream files where they can, so few
upstream lines move. The doc citations they did move (config.rs,
builder.rs, driver/mod.rs) are corrected. The doc-citation guard reports
no drift beyond what main already has.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BXj7C4ZeFuczxxDVadAHZ5
…estart, liveness, config - P1: a link that came up over TCP may have negotiated far more than one UDP datagram carries. Both ends lower the link MTU to DIRECT_LINK_MTU (1400, as rns-rs does) when the link moves onto the direct interface, and the interface reads datagrams up to 64 KiB, so a part already in flight at the old size is not truncated. - A full application sink no longer holds back signals. If the next in-order channel message is a direct-link signal it is admitted, and signals stop being charged to the delivery budget. - When a link's relayed fallback interface goes down while the link is direct, the fallback is forgotten. If the direct path then dies too, the link closes as Stale instead of being pinned to a removed index. - NodeCore::abandon_direct_links, called when the event loop starts, drops sessions and direct attachments left from a previous run of the node (stop, then start). - The direct interface counts a punch frame as liveness only once its session and token check out. - An IPv6-only facilitator resolves (IPv4 still preferred). A malformed INI probe_port is a config error instead of quietly running no facilitator. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BXj7C4ZeFuczxxDVadAHZ5
…re, recover from one-way failure
- P2: a full channel window or channel pacing no longer fails the upgrade.
The signal waits in a per-link outbox (in order, at most 8) and is
flushed on every tick. The session's own phase timer still bounds the
wait. Only a dead link or a hard channel error ends it.
- P1: one-way failure of the direct path. Two things were wrong. The two
ends could end up on different paths. And on a fast link the link went
stale (about 20 s) long before the direct interface's 120 s silence
timer, so the fallback never got to run. Now:
- a link that goes stale while on a direct interface falls back at
once: the direct interface is retired, and a keepalive goes out over
the relay;
- a link packet arriving over the relay the link left, more than
FALLBACK_GRACE_MS (10 s) after the upgrade, means the peer has fallen
back, so this end follows (interface retired, keepalive sent on the
relay).
Both ends converge on the relay, and the stale link recovers instead of
closing.
Tests: the backpressure test saturates the channel and asserts the
ACCEPT was held and then flushed. The stale-fallback test checks the
nudge goes out on the relay. The return test checks a packet inside the
grace is ignored and one after it brings the end back.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BXj7C4ZeFuczxxDVadAHZ5
…wide session ids, repeated announces - Fallback now triggers only on authenticated traffic. Leaving the direct path because of a packet over the relay was decided before the packet was validated, so anyone able to inject onto the relay could pull a node off a working direct link. Most link handlers record liveness before they decrypt, so liveness is no evidence. The trigger is now a channel packet that has decrypted under the link key (direct_link_authenticated, fed by the receiving interface that process_link_packet notes). The fallback nudge is therefore authenticated too: besides the keepalive, which helps a stale link recover but proves nothing, it sends COMPLETE for the session that made the link direct. The peer ignores COMPLETE as a protocol message (it no longer holds that session) and acts on where it arrived from. Signals may go out on a Stale link. - A REQUEST whose session id is already in use anywhere on the node is refused as BUSY, not just one on the same link. The driver keys sockets and tasks by session id, so a reused id could take over another link's probe. - A new direct interface repeats its announces 1 s and 4 s after registration. The first can go out while the peer is still inside its punch loop, which discards everything but punch frames. That would leave the peer without a route to this node's destinations. Tests: a forged keepalive and a tampered channel packet on the relay leave the link direct, while the stale end's nudge alone brings the peer back; a reused session id starts no second probe; the repeat schedule fires for live interfaces only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BXj7C4ZeFuczxxDVadAHZ5
…box, orphaned interface, stale recovery - A signal buffered behind a sequence gap now drains even when the host sink is full (Channel::drain_received_budgeted, in a child module so channel/mod.rs moves no cited line). Before, only an in-order signal got past the full-sink refusal, so an ACCEPT or READY behind a gap sat unhandled and unproved until the application drained, and the upgrade timed out. Order still holds: draining stops at the first application message once the limit is spent. - When a session ends, its signals still waiting in the outbox go with it. Before, a REQUEST that waited out its proposal timeout was sent once the channel opened, and the peer started on a session nobody held. - A link that closes with no fallback left also retires its direct interface. The attachment was already removed, so remove_link could not find the interface, and the punched socket lived on while the peer kept it alive. - A punch that lands on a link that went stale meanwhile makes it Active again and emits LinkRecovered. Before, it stayed Stale and the stale watchdog closed a working link. Tests: the budgeted drain past a spent limit, with order preserved; an expired session's held REQUEST is dropped; an orphaned direct interface is retired; a stale link recovers on attach. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BXj7C4ZeFuczxxDVadAHZ5
…sfers finish first, signals past a full sink, terminal REJECT kept - P1: every fallback to the relay now sends the peer the authenticated nudge (keepalive plus COMPLETE), including when the direct interface's own inactivity timer fires first. Before, that path moved only this end back. The peer stayed on its working direction, kept alive by unauthenticated keepalives over the relay, while this end's link went stale and closed. - A punched link with a resource transfer under way, in either direction or in off-lock assembly, stays on the relay until the transfer is done, then moves on the next tick. A transfer cut for the link's earlier, larger MTU can carry parts no UDP datagram can. If the interface dies or the node restarts while the link waits, the upgrade is reported failed. - A full host sink no longer refuses any direct-link signal, whatever its sequence. A duplicate of a delivered one is re-proved, so a lost proof no longer costs the sender its channel retries. Application messages are still refused. - When a session ends, a REJECT queued in the outbox stays. It is the session's last word, and the initiator should hear it rather than time out. Tests: a dead direct interface nudges over the relay; a punched link waits out a transfer and then moves; a full sink re-proves a duplicate REQUEST; a responder's queued REJECT outlives its session. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BXj7C4ZeFuczxxDVadAHZ5
… telemetry (#77), the close-during-assembly pin Lands PRs #74, #75 and #76. - #74: direct-link upgrade, wire-compatible with rns-rs. Five Codex rounds. - #75: a test pinning the close-during-assembly invariant CIRISEdge relies on. - #76: link census, link list, lifecycle counters, the mirror-divergence alarm and the metrics facade. Four Codex rounds. The final fixes on #74 (round 5) and #76 (reject_link) answer Codex's own findings. Codex then hit its usage limit and could not review them; each has a test. Releasing on green CI was the maintainer's call. Integration: #74 and #76 overlapped only where both appended to the same places (NodeCore fields, ReticulumNode fields, lib.rs re-exports, the event loop's prologue and timer branch, the lock census). Each commit builds on its own. The apply_inbound citation in event_log.rs follows the combined driver. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019ye3GBY8MB38EsYH5eaZKq
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 84e41c1b77
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| let Some(link) = self.links.get_mut(link_id) else { | ||
| return; |
There was a problem hiding this comment.
When an outgoing link establishes after an establishment-timeout retry, callers still hold the original ID while the live entry is stored under the re-keyed wire ID. Although this method now promises to close links initiated by this node, the direct self.links.get_mut(link_id) lookup returns None for that documented case, so reject_link silently leaves the link open and emits no LinkClosed; resolve the alias as close_link does before performing this lookup.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in bdd1184. reject_link now runs resolve_link_id first, as close_link does, so an original ID handed out before an establishment retry reaches the re-keyed link and closes it with LinkClosed.
| Some((PunchKind::Ack, _)) => acked_by = Some(src), | ||
| None => continue, |
There was a problem hiding this comment.
One endpoint can register its direct interface and emit DirectLinkEstablished while the peer is still waiting for the final punch ACK; the REANNOUNCE_AFTER comment explicitly accounts for this overlap. If the established endpoint immediately sends an ordinary Reticulum datagram, this peer consumes it from the shared socket here and silently drops it as None; the announce repeats repair only announces, not one-shot packets such as link-close or identify traffic. Buffer or forward these datagrams into the eventual interface, or delay the established handoff until both peers are known to have exited punching.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in bdd1184, with the buffering option. punch() now keeps ordinary datagrams from the peer that arrive before it finishes (up to 64; its buffer was 128 bytes and is now full UDP size). They travel in Outcome::Punched, and the new direct interface delivers them before anything it reads itself. Datagrams arriving after the punch and before the interface task starts wait in the socket's kernel buffer. Test: datagrams_the_peer_sends_during_the_punch_are_kept.
| /// Direct-link upgrades (+ciris, leviculum#70): sessions, the driver's | ||
| /// job queue, and links moved onto punched interfaces. | ||
| direct_links: direct_link::DirectLinks, |
There was a problem hiding this comment.
The new DirectLinks state owns several heap-backed maps and queues (sessions, attached, deferred, per-link outboxes, and jobs), but NodeCore::heap_census accounts for none of them; node_struct includes only the inline container headers. Consequently the newly published leviculum.memory.bytes metrics under-report memory whenever upgrades or established direct links exist, undermining the census specifically when operators use it to diagnose link-related heap growth. Add these backing allocations to an appropriate census component.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in bdd1184. NodeCore::heap_census now adds direct_link_heap_bytes() (sessions, proposal times, attachments, deferred links, outboxes and the job queue) to its links component, which feeds leviculum.memory.bytes{component="links"}. Test: the_heap_census_counts_direct_link_state.
| if self.resources_in_flight(&link_id) { | ||
| // A transfer cut for the link's present MTU may | ||
| // carry parts no UDP datagram can (Codex review | ||
| // on #74): the link moves once it is done. |
There was a problem hiding this comment.
When a link negotiated a large MDU over a carrier such as TCP, an unproved channel envelope can still be outstanding when punching succeeds, but this gate defers only resource transfers. The attach then lowers the link MTU and moves retransmissions onto UDP; an envelope near the channel's valid 65,535-byte limit produces an encrypted packet larger than UDP can send, so send_to repeatedly fails while channel retries are still charged, eventually tearing down an otherwise working link. Include outstanding channel traffic in the deferral condition before switching carriers.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in bdd1184. The deferral now also covers the channel. Channel::outstanding_exceeds_mtu(DIRECT_LINK_MTU) holds a punched link on the relay while any unproved envelope is larger than the channel can send at the direct MTU. Once it's proved, the next tick moves the link, exactly as for an in-flight resource. Test: a_channel_message_too_big_for_udp_holds_the_link_on_the_relay (link at a 16384 MTU, a 4000-byte message outstanding, no move until it's proved, then the move).
…tch, early datagrams kept, direct-link heap counted, reject_link resolves re-keyed ids - P1: a channel message still awaiting its proof that is larger than the channel can send at DIRECT_LINK_MTU now holds a punched link on the relay, as an in-flight resource already did. Its retransmissions would not fit a UDP datagram, and the link would die of channel retries (Channel::outstanding_exceeds_mtu). - Ordinary datagrams the peer sends while this end is still punching are kept, up to 64, and delivered first by the new direct interface. The peer can finish first and send at once, and one-shot packets (a close, an identify) are not repeated. The punch buffer is now full UDP size; it was 128 bytes. - The heap census counts the direct-link bookkeeping (sessions, attachments, deferred links, outboxes, jobs) in its links component. - reject_link resolves a re-keyed id first, as close_link does. A link re-keyed by an establishment retry lives under its new wire id, and callers hold the original. Tests: an oversized outstanding message defers the move until it is proved; early datagrams survive the punch; a running session shows in the census. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019ye3GBY8MB38EsYH5eaZKq
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: bdd1184861
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| let busy = self.direct_links.sessions.contains_key(&link_id) | ||
| || self.direct_links.interface_of_link(&link_id).is_some() | ||
| || self | ||
| .direct_links | ||
| .sessions | ||
| .values() | ||
| .any(|s| s.id() == &session); |
There was a problem hiding this comment.
When an allowed peer sends a new REQUEST after its previous session finishes, busy becomes false immediately because the cooldown map is checked and updated only by the local propose_direct_link path. A buggy or malicious peer can therefore repeatedly make this node bind sockets and send probes to a peer-selected facilitator, once per failed session on every link, bypassing the stated 60-second protection. Apply the per-link cooldown before accepting an incoming REQUEST as well.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 106cf79. An incoming REQUEST inside the link's 60 s cooldown is refused with REJECT_BUSY, and accepting one records the time, so both directions share one cooldown per link. Test: a_peer_cannot_propose_again_inside_the_cooldown.
| && !frame.is_empty() | ||
| && early.len() < EARLY_MAX | ||
| { | ||
| early.push(frame.to_vec()); |
There was a problem hiding this comment.
During punching, an allowed peer can send 64 non-punch datagrams of up to 65,535 bytes from its advertised endpoint, and each is cloned here, retaining about 4 MiB per concurrent session until success or the 10-second timeout. With multiple accepted links this becomes a straightforward memory-exhaustion path; legitimate packets sent after direct attachment are constrained by the 1,400-byte direct MTU, so reject oversized early packets or impose a small aggregate byte budget rather than limiting only the item count.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 106cf79. Early datagrams are now kept only if each is at most 2048 bytes (anything bigger isn't a packet from a link at DIRECT_LINK_MTU) and the total stays within 64 KiB, as well as the 64-datagram cap. Test: early_datagrams_are_bounded_by_size.
| self.events.push(NodeEvent::DirectLinkLost { | ||
| link_id: attachment.link_id, | ||
| interface_index: index, | ||
| }); |
There was a problem hiding this comment.
When an outgoing link was re-keyed during establishment and its direct path later dies without a fallback, this event is queued with the wire ID, after which the None branch removes the link and emit_link_closed consumes the link_origin_ids entry. By the time process_events_and_actions rewrites queued events, the mapping is gone, so DirectLinkLost exposes the unexpected wire ID while the following LinkClosed uses the original caller-visible ID. Translate this event before consuming the mapping or retain the mapping through the event drain.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 106cf79. release_direct_attachment translates the ID through link_origin_ids before pushing DirectLinkLost, so the event carries the caller-visible ID even when the following close consumes the mapping.
| if attachment.previous != Some(iface) | ||
| || now_ms < attachment.attached_at_ms.saturating_add(FALLBACK_GRACE_MS) | ||
| { |
There was a problem hiding this comment.
If one endpoint loses or explicitly removes its direct interface during the first 10 seconds after attachment, it immediately sends the authenticated Complete nudge over the relay, but this condition discards that packet solely because it arrived during the grace period. On a low-RTT channel the nudge's retry budget can be exhausted before the grace expires, leaving the peer pinned to the dead direct path until its stale or 120-second interface-inactivity timeout. Recognize the authenticated fallback signal during the grace period or schedule a repeat after the grace ends.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 106cf79. A COMPLETE for the attaching session that arrives over the saved relay now triggers fallback immediately (direct_link_nudged), grace or not. Ordinary relayed traffic still waits out the grace. Test: a_nudge_inside_the_grace_still_brings_the_peer_back (a nudge at +1 s).
| self.links.get(link_id).is_some_and(|link| { | ||
| link.outgoing_resource().is_some() | ||
| || link.incoming_resource().is_some() | ||
| || link.channel().is_some_and(|ch| { |
There was a problem hiding this comment.
On a link using application-mediated resource acceptance, the receiver can hold a pending_resource_adv while both incoming_resource() and outgoing_resource() are None, so this predicate lets it attach and lower its MTU even though the sender remains deferred with the outgoing transfer. If the advertisement was created on a large-MTU carrier, a later accept_resource recomputes a larger part count from the 1,400-byte MTU while the sender still has the old, larger parts, leaving the receiver waiting for parts that do not exist. Treat has_pending_resource() as in-flight state and keep both endpoints on the relay until the advertisement is accepted or rejected.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 106cf79. has_pending_resource() is now part of the in-flight predicate, so a punched link stays on the relay (deferred) until the advertisement is accepted or rejected. Test: a_pending_resource_advertisement_holds_the_link_on_the_relay.
| let attached = core::mem::take(&mut self.direct_links.attached); | ||
| for (index, attachment) in attached { | ||
| self.release_direct_attachment(index, attachment, "node restarted", false); |
There was a problem hiding this comment.
On a stop()/start() cycle, the old runtime drops direct-interface tasks after the event loop has stopped, so the core never receives handle_interface_down for their indices. This restart cleanup restores attached links to their fallback but does not remove paths, names, or other transport state associated with the abandoned direct index; in particular, the one-hop destination path learned during attachment can keep routing new traffic to a nonexistent or later-reused interface. Run the normal interface-down cleanup for each abandoned direct interface before releasing the attachment.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 106cf79. abandon_direct_links now runs handle_interface_down for every old direct interface, attached or deferred. That removes its paths, name and other transport state, and also moves the link back (or fails a deferred one). Test: a_restart_takes_the_old_direct_interface_down_in_full.
| for attachment in self.direct_links.attached.values_mut() { | ||
| if attachment.previous == Some(index) { | ||
| attachment.previous = None; | ||
| } |
There was a problem hiding this comment.
If the relay interface goes down while a punched link is deferred for an in-flight transfer, this loop clears that index only from already-attached links; the deferred record carries no fallback state and the link itself remains attached to the now-dead relay index. Once the transfer finishes, attach_direct_link captures that dead index as previous, so a later direct-path loss restores the link to an interface that no longer exists instead of closing it as documented. Track and clear the fallback for deferred links as well.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 106cf79, at the root rather than in the deferred record. At attach, previous is kept only if that interface still exists in transport (interface_name(prev).is_some()). That covers a relay lost while the link was deferred and also one lost while the punch ran, since handle_interface_down never repoints links. Test: a_fallback_that_died_before_the_move_is_no_fallback.
…ooldown, nudge in grace, dead fallbacks, restart cleanup, early-datagram bytes - P1: a resource advertisement waiting for the application now holds a punched link on the relay, like a transfer under way. Accepting it later sizes its parts from the link's MTU, so moving first would leave the receiver waiting for parts the sender cut larger. - The 60 s proposal cooldown now binds the peer too. A REQUEST inside it is refused as BUSY, and accepting one starts the clock. Before, a peer could keep this node binding sockets and probing a facilitator of its choosing. - The fallback nudge (COMPLETE for the attaching session, arriving over the relay) is acted on during the 10 s grace as well. It is unambiguous, unlike ordinary relayed traffic, and a peer that lost its path early was otherwise ignored until the nudge's retries ran out. - A fallback interface that went down before the move, whether while the punch ran or while the link waited out a transfer, is not kept as the fallback. A later loss of the direct path closes the link instead of repointing it at a dead index. - On restart, abandon_direct_links takes each old direct interface down through handle_interface_down. Its paths and the rest of its transport state go with it, not just the link's attachment. - DirectLinkLost carries the id the application holds even when the link closes in the same step and the re-key mapping is consumed. - Early datagrams kept during a punch are bounded by size as well as count: at most 2048 bytes each and 64 KiB in all. Tests for each except the re-key id path (it needs an establishment retry): pending advert holds, cooldown refuses, nudge inside the grace, dead fallback closes, restart forgets the interface, early datagrams bounded. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019ye3GBY8MB38EsYH5eaZKq
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
…tch, early datagrams kept, direct-link heap counted, reject_link resolves re-keyed ids - P1: a channel message still awaiting its proof that is larger than the channel can send at DIRECT_LINK_MTU now holds a punched link on the relay, as an in-flight resource already did. Its retransmissions would not fit a UDP datagram, and the link would die of channel retries (Channel::outstanding_exceeds_mtu). - Ordinary datagrams the peer sends while this end is still punching are kept, up to 64, and delivered first by the new direct interface. The peer can finish first and send at once, and one-shot packets (a close, an identify) are not repeated. The punch buffer is now full UDP size; it was 128 bytes. - The heap census counts the direct-link bookkeeping (sessions, attachments, deferred links, outboxes, jobs) in its links component. - reject_link resolves a re-keyed id first, as close_link does. A link re-keyed by an establishment retry lives under its new wire id, and callers hold the original. Tests: an oversized outstanding message defers the move until it is proved; early datagrams survive the punch; a running session shows in the census. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019ye3GBY8MB38EsYH5eaZKq
Release PR for v0.30.0+ciris.1. It lands three reviewed PRs as their own commits, plus a release commit.
metricsfacadeNot reviewed by Codex: the final fixes on #74 (round 5,
ed7e5d00) and #76 (reject_link,a4b25547). They answer Codex's own findings, and each has a test. Codex then reached its usage limit and could not review them. Releasing once CI is green is the maintainer's call.Integration: #74 and #76 overlapped only where both append to the same places:
NodeCoreandReticulumNodefields;lib.rsre-exports;The conflicts were resolved by keeping both sides. The two that landed mid-function were folded into the commits where they arose, and every commit builds on its own.
Local gate on the release head:
-D warnings) clean;direct_link_upgradeandtelemetry_publishpass;cargo denyand THIRD-PARTY-NOTICES current;Merge plan: rebase-merge once CI is green, tag
v0.30.0+ciris.1on the merge, thengh release create --verify-tag, and close #74/#75/#76 as landed here.🤖 Generated with Claude Code
https://claude.ai/code/session_019ye3GBY8MB38EsYH5eaZKq