feat: add collector-aware as2rel output alongside classic aggregate - #13
Merged
Conversation
Add as2rel-collector-index and pfx2as-collector-index binaries that preserve per-collector provenance in latest-snapshot aggregates. - as2rel-collector-index: walks daily as2rel_*.bz2 files and produces as2rel-collector-latest.json.bz2 with per-edge collector breakdown - pfx2as-collector-index: same pattern for pfx2as data - Each output includes generated_at timestamp, input file count, total/cross-collector sums, and collector-level detail - Fix pre-existing clippy warnings in bootstrap.rs (useless borrows in format! macros)
The previous implementation used HashMap<(u32,u32,u8), HashMap<String,Detail>> which created per-edge sub-HashMaps causing 30GB+ memory usage with ~500K edges. Now uses: - Flat Vec<(u32,u32,u8,u16,usize,usize)> for as2rel - Flat Vec<(u32,u32,u16,usize)> for pfx2as (with prefix string interning) - Sort by key then group with O(1) extra memory - Manual streaming JSON writer avoids serde_json::to_value() DOM tree
Rewrite as2rel-index to produce BOTH outputs from a single file pass:
1. Classic: {prefix}-latest.json.bz2 (unchanged v1 format — no breaking change)
2. Collector: {prefix}-collector-latest.json.bz2 (new per-collector provenance)
Uses flat vector + sort approach for memory efficiency (~1.7GB peak RSS
for 21M records). All three prefixes (as2rel, as2rel-v4, as2rel-v6) get
both output files.
Remove standalone as2rel-collector-index and pfx2as-collector-index
binaries — collector output is now integrated into as2rel-index.
This was referenced Jul 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Extends
as2rel-indexto produce per-collector provenance data alongside the existing classic aggregate, from a single pass over daily files.Changes
as2rel-indexnow produces TWO output files per prefix (as2rel, as2rel-v4, as2rel-v6):{prefix}-latest.json.bz2— identical format to v1, no breaking changes{prefix}-collector-latest.json.bz2— per-edge collector breakdown with per-collector paths/peers countsMemory-efficient flat vector + sort approach — ~1.7GB peak RSS for 21M records (previous nested-HashMap approach used 30GB+)
Removed standalone
as2rel-collector-indexandpfx2as-collector-index(pfx2as collector deferred)Output format (collector file)
{"generated_at":"2026-07-23T17:22:12Z","input_files":79,"entries":[ {"asn1":13335,"asn2":174,"rel":1, "total_paths_count":120,"total_peers_count":12,"collector_count":14, "collectors":{ "route-views3":{"project":"route-views","paths_count":42,"peers_count":3}, "rrc00":{"project":"riperis","paths_count":35,"peers_count":2} } } ]}Tested