Repository navigation
Conversation
…e#819, leviculum#57) A field gauge for link accumulation. NodeCore::link_census, and ReticulumNode::link_census in the driver, return the established links (Active or Stale) split into initiator and responder, a count of pending ones, and the same split per destination, most links first. It names the owner of a growing count. The initiator keeps a link alive with its keepalives, so a rising initiator count toward one destination means this node is not closing what it dialled. A rising responder count under its own destination means the peers are not closing theirs. CIRISEdge#819 traced a canonical's 1024 links per 35 min to the first case: its dial pool grows to demand and never shrinks. Before that, diagnosing it took a reading of the reap rules from the source. One walk of the link table under a brief core lock. Added to the lock census, together with known_destination_evictions (leviculum#49), which should have been added when that method landed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BXj7C4ZeFuczxxDVadAHZ5
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7d2ae854fc
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| /// leviculum#57). A field gauge for link accumulation: see | ||
| /// [`leviculum_core::node::NodeCore::link_census`] for how to read it. | ||
| /// Takes the core lock briefly; walks the link table once. | ||
| pub fn link_census(&self) -> leviculum_core::node::LinkCensus { |
There was a problem hiding this comment.
For applications that depend only on leviculum-std, this signature exposes a type they cannot name: Cargo does not make the transitive leviculum_core crate available to downstream code, and leviculum-std/src/lib.rs omits both LinkCensus and DestinationLinks from its core-type re-exports. Calls using inference work, but consumers cannot use the result in annotations, struct fields, or APIs without adding a separate core dependency, unlike existing results such as LinkStats and TransportStats; re-export both new types from leviculum_std.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 24dca1c. These are now all re-exported from leviculum_std, next to LinkStats and TransportStats: LinkCensus, DestinationLinks, plus the telemetry types this PR has since grown (LinkInfo, LinkLifecycle, LinkRole, LINK_CLOSE_REASONS, link_close_reason_name, LinkState, NodeHeapCensus). The telemetry_publish integration test now names every result through leviculum_std alone.
… facade (leviculum#77) The P0 and P1 leviculum asks of CIRISServer's unified-telemetry FSD (PR #746, §4). The maintainer has made instrumentation the priority before the next scale step. P0 - ReticulumNode::link_list (NodeCore::link_list): every link with its caller-visible id, destination, role, state, age, idle seconds, RTT and interface. - Cumulative lifecycle counters, link_lifecycle: established by role, closed by LinkCloseReason, and handshake_failed by reason for links that never established. remove_link notes whether the departing link had established, and emit_link_closed counts it. Every close follows its removal, so established - closed equals the live census. - Divergence alarm. Every 10 s the event loop compares the completion mirror with the core's established count (MirrorWatch). A disagreement that outlasts 30 s raises one LINK_MIRROR_DIVERGED event (catalogued) per episode, counted in link_count_check().alarms. - The per-destination census from the previous commit. P1, facade per FSD §5: `metrics` in the library, the exporter in the host - The `metrics` crate (default-features off; its one dependency is rapidhash). ReticulumNode::publish_metrics records the lifecycle, the census, the mirror check, TransportStats (drops by every DropReason, through the new TransportStats::drops_for), PlaneStats, the known- destination evictions and the core's heap census. That last one gives std hosts per-component memory and a mean bytes-per-link. It is pull based: the host calls it on its scrape cadence. Counters use absolute(), so a scrape reads the node's own totals. - telemetry::METRIC_CATALOG lists 24 metrics, each with a kind, a unit, its allowed label keys and a description; describe_* is registered from it once. Every label value is a bounded enumeration. No link, destination or peer id is ever a label. - event_catalog_completeness now scans metric sites too. Every recorded name must be catalogued, every catalogued name recorded, and every label key allowed. The source walk is shared with the event scan. Housekeeping - THIRD-PARTY-NOTICES regenerated for metrics and rapidhash. This also picks up a stale rustls version line that was already wrong on main. - leviculum-nrf/Cargo.lock now carries the fork's version string, which the release bumps never updated. cargo-about --frozen failed on it. - The lock census adds the new methods (TOTAL 65). - event_log.rs's apply_inbound citation already pointed at unrelated code on main; it is corrected. - The doc-citation guard reports no drift beyond what main already has. Tests: mvr_link_census (list, counters by reason, handshake failure, and the counters agreeing with the census); drops_for over every reason; MirrorWatch (grace, one alarm per episode, re-arm); the metric scanner; and telemetry_publish, which runs two real nodes over TCP through the whole life of a link with a capturing recorder. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BXj7C4ZeFuczxxDVadAHZ5
…td (Codex on #76) LinkCensus, DestinationLinks, LinkInfo, LinkLifecycle, LinkRole, LINK_CLOSE_REASONS, link_close_reason_name, LinkState and NodeHeapCensus are now nameable from leviculum_std, as LinkStats and TransportStats already were. A consumer without its own leviculum-core dependency can use them in annotations and fields. telemetry_publish names every result through leviculum_std alone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BXj7C4ZeFuczxxDVadAHZ5
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 24dca1c770
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if let Some(link) = self.links.get(link_id) { | ||
| let established = link.established_at_secs().is_some(); | ||
| self.note_link_removing(established); |
There was a problem hiding this comment.
When an application calls reject_link for a pending inbound link, this records was_established = false, but that path only invokes remove_link and never the emit_link_closed call that consumes the marker and increments handshake_failed. Consequently, rejected handshakes disappear from the cumulative lifecycle metrics even though LinkLifecycle promises every link that ends before establishment is counted; update the rejection path to record an appropriate failure reason.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 7e33a81. reject_link now counts the refused request as LinkLifecycle::rejected (it is reported as leviculum.link.handshake_failed{reason="rejected"}) and clears the removal marker. The census test checks that a refused request counts on the refusing node.
| fn describe_all() { | ||
| DESCRIBED.call_once(|| { | ||
| for spec in METRIC_CATALOG { |
There was a problem hiding this comment.
If publish_metrics() is first called before a global recorder is installed, or with a temporary local recorder, this process-wide Once is consumed while descriptions are sent to that recorder (or the no-op recorder). A subsequently installed exporter still receives metric values but never receives the catalogue's units and descriptions, and separate local recorders have the same problem. Description registration needs to be tied to the active recorder rather than permanently suppressed after the first call.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 7e33a81. The Once is gone: every publish registers each catalogue entry's unit and description with whatever recorder is installed at that moment (24 calls per publish). telemetry_publish checks that a recorder installed after earlier publishes receives every description.
| age_secs: established.map(|at| now_secs.saturating_sub(at)), | ||
| idle_secs: established | ||
| .map(|_| now_secs.saturating_sub(link.last_inbound_secs())), | ||
| rtt_ms: link.rtt_ms(), |
There was a problem hiding this comment.
For every pending or handshake link included by link_list, Link::rtt_ms() returns CHANNEL_DEFAULT_RTT_MS (500 ms) when no RTT has been measured, so this field reports a fabricated measurement during establishment; a sub-millisecond initiator link can also retain this fallback because its zero measurement is not stored. This is inconsistent with the field's “Measured round-trip time” contract and the existing LinkStats::rtt_ms: Option<u64> API, and can make operators diagnose high latency where no sample exists; represent the unmeasured state explicitly.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 7e33a81. LinkInfo::rtt_ms is now Option<u64> (rtt_us()/1000), None until a sample exists, matching LinkStats::rtt_ms.
| counter!("leviculum.link.established", "role" => "initiator") | ||
| .absolute(lc.established_initiator); | ||
| counter!("leviculum.link.established", "role" => "responder") | ||
| .absolute(lc.established_responder); |
There was a problem hiding this comment.
When a process replaces a ReticulumNode while retaining its recorder, or publishes metrics for two nodes, each node writes its independent since-start total into the same unlabeled series. The metrics 0.24 contract requires CounterFn::absolute to keep at least the supplied value, so a replacement node's counters remain stuck at the previous node's totals until they exceed them, while concurrent nodes expose only the maximum rather than an aggregate. Give node instances distinct series or maintain process-level cumulative offsets before using absolute.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 7e33a81, a little differently from either option. Plain increments would have left a recorder installed after earlier publishes without the totals. So each node keeps the totals it last published (CounterBaseline, held by ReticulumNode) and adds its growth to one process-wide sum per series, and absolute() records that sum. Two nodes in one process sum, a replacement node carries on from where the series stood, and a late recorder still reads full totals. Gauges stay last-publish-wins, which is documented. telemetry_publish checks that two nodes sum into one recorder.
…is None, describes per publish, process-total counters
- reject_link counts the refused request, as LinkLifecycle::rejected and
as leviculum.link.handshake_failed{reason="rejected"}. remove_link
noted the departing link but reject_link reports no close, so refused
handshakes vanished from the metrics.
- LinkInfo::rtt_ms is Option<u64>: None until a sample exists, as in
LinkStats. Before, a pending link reported the 500 ms working fallback
as though it had been measured.
- Units and descriptions are registered on every publish instead of once
per process, so a recorder installed (or swapped for a local one) after
the first publish still learns them.
- Counters are process totals. Each node keeps the totals it last
published (CounterBaseline, held by ReticulumNode) and adds its growth
to one process-wide sum per series, which absolute() records. Two nodes
in one process now sum, and a replacement node carries on instead of
sitting under its predecessor's total, which absolute's keep-the-
maximum contract had caused. A recorder installed late still reads full
totals. Plain per-node increments would have lost exactly that.
Tests: the census test covers a refused request. telemetry_publish checks
that a late recorder receives every description, that two nodes sum into
one recorder, and that an established link has a measured RTT.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BXj7C4ZeFuczxxDVadAHZ5
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7e33a81578
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| let existed = self.links.contains_key(link_id); | ||
| self.remove_link(link_id); | ||
| if existed { | ||
| self.note_link_rejected(); |
There was a problem hiding this comment.
When reject_link is called with any existing link ID, including an already-established or locally initiated link, this path removes the link and records it as rejected. Because note_link_rejected also clears the removal marker and no close is emitted, rejecting an established link permanently breaks the documented established - closed_total() == live links invariant. Check that the link is a pending responder before classifying it as an application-rejected request, or account other removed links as closures.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in a4b2554, and it was worse than the counters. Dropping a live link with no LinkClosed also leaves the driver's completion mirror holding it, which is the very divergence the alarm exists to catch. reject_link now refuses only a pending inbound request (not the initiator, never established), counted as rejected. Any other link is closed the way close_link closes it: close packet, LinkClosed { reason: Normal }, counted as a close. The census test rejects a live link and checks that it counts as a close and that established - closed_total still equals the census.
…ng else is closed (Codex on #76) reject_link removed whatever link it was given, without a close. On an established or locally dialled link, that left the completion mirror and the lifecycle counters believing the link alive, and broke established - closed == live. Now: - a pending inbound request is refused and counted as rejected; - any other link is closed the way close_link closes it (close packet, LinkClosed with reason Normal, counted as a close). Its events go out with the node's next output. Test: rejecting a live link counts a close, not a refusal, and the counters still agree with the census. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BXj7C4ZeFuczxxDVadAHZ5
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
…td (Codex on #76) LinkCensus, DestinationLinks, LinkInfo, LinkLifecycle, LinkRole, LINK_CLOSE_REASONS, link_close_reason_name, LinkState and NodeHeapCensus are now nameable from leviculum_std, as LinkStats and TransportStats already were. A consumer without its own leviculum-core dependency can use them in annotations and fields. telemetry_publish names every result through leviculum_std alone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BXj7C4ZeFuczxxDVadAHZ5
…is None, describes per publish, process-total counters
- reject_link counts the refused request, as LinkLifecycle::rejected and
as leviculum.link.handshake_failed{reason="rejected"}. remove_link
noted the departing link but reject_link reports no close, so refused
handshakes vanished from the metrics.
- LinkInfo::rtt_ms is Option<u64>: None until a sample exists, as in
LinkStats. Before, a pending link reported the 500 ms working fallback
as though it had been measured.
- Units and descriptions are registered on every publish instead of once
per process, so a recorder installed (or swapped for a local one) after
the first publish still learns them.
- Counters are process totals. Each node keeps the totals it last
published (CounterBaseline, held by ReticulumNode) and adds its growth
to one process-wide sum per series, which absolute() records. Two nodes
in one process now sum, and a replacement node carries on instead of
sitting under its predecessor's total, which absolute's keep-the-
maximum contract had caused. A recorder installed late still reads full
totals. Plain per-node increments would have lost exactly that.
Tests: the census test covers a refused request. telemetry_publish checks
that a late recorder receives every description, that two nodes sum into
one recorder, and that an established link has a measured RTT.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BXj7C4ZeFuczxxDVadAHZ5
…ng else is closed (Codex on #76) reject_link removed whatever link it was given, without a close. On an established or locally dialled link, that left the completion mirror and the lifecycle counters believing the link alive, and broke established - closed == live. Now: - a pending inbound request is refused and counted as rejected; - any other link is closed the way close_link closes it (close packet, LinkClosed with reason Normal, counted as a close). Its events go out with the node's next output. Test: rejecting a live link counts a close, not a refusal, and the counters still agree with the census. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BXj7C4ZeFuczxxDVadAHZ5
… telemetry (#77), the close-during-assembly pin Lands PRs #74, #75 and #76. - #74: direct-link upgrade, wire-compatible with rns-rs. Five Codex rounds. - #75: a test pinning the close-during-assembly invariant CIRISEdge relies on. - #76: link census, link list, lifecycle counters, the mirror-divergence alarm and the metrics facade. Four Codex rounds. The final fixes on #74 (round 5) and #76 (reject_link) answer Codex's own findings. Codex then hit its usage limit and could not review them; each has a test. Releasing on green CI was the maintainer's call. Integration: #74 and #76 overlapped only where both appended to the same places (NodeCore fields, ReticulumNode fields, lib.rs re-exports, the event loop's prologue and timer branch, the lock census). Each commit builds on its own. The apply_inbound citation in event_log.rs follows the combined driver. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019ye3GBY8MB38EsYH5eaZKq
Requested from the CIRISEdge side after CIRISEdge#819: a canonical running v0.27 grew about 1024 established links per 35 minutes. Reading the reap rules from source showed nothing in leviculum leaks: a silent link closes within about 13 minutes. So the links were being kept alive by their owner. Edge traced it to its dial pool, which grows to demand and never shrinks, while the pooled idle links are kept alive by edge's own keepalives. That fix is edge-side. This PR makes the same diagnosis readable from a running node next time.
API
NodeCore::link_census()andReticulumNode::link_census()return aLinkCensuswith:initiator: established links (Active or Stale) this node dialled;responder: established links peers opened to this node;pending: links still establishing;by_destination: the same split per destination, most links first.Reading it: the initiator keeps a link alive with its keepalives.
Cost: one walk of the link table under a brief core lock. Added to the lock census (
scripts/core-lock-census.txt), together withknown_destination_evictionsfrom #49, which was missed when it landed. Theifac_*census failure is still there, unchanged.Tests:
mvr_link_censuscovers a 2-established/1-pending pair seen from both ends, and a closed link leaving the census. Core lib 2289 and std lib 877 pass; clippy is clean; thumbv6m and thumbv7em build. The doc-citation guard reports no drift beyond what main already has.This likely explains #57 (2:1 established:closed), which reports the same symptom.
🤖 Generated with Claude Code
https://claude.ai/code/session_01BXj7C4ZeFuczxxDVadAHZ5