feat: add Cisco sh ip bgp text dump parser with unified MRT/text API - #321
Conversation
Port the fixed-width text RIB dump parser from monocle (#143, #146) into bgpkit-parser so it can be used programmatically. The new parser::text_dump module handles PCH daily snapshots and route-views oix-full-snapshot files, extracting column offsets from the table header, handling multipath continuations and wrapped prefixes (including IPv6), and tolerating missing preambles (route-views sentinel peer identity). Add unified BgpkitParser constructors so text dumps integrate into the standard for-elem-in-parser loop: - new_text(path) / from_text_reader(r): parse a known text dump - new_auto(path) / from_auto_reader(r): peek first bytes, auto-dispatch text vs MRT Text-dump elements are materialized at construction (RIB snapshots are full snapshots, not streaming). All filter methods work on both paths. The default new(path) constructor remains MRT-only. 22 unit tests + 5 constructor integration tests cover detection, header parsing, continuations, wrapped prefixes, origin codes, route-views sentinel behavior, timestamp inference, auto-detection, and filter integration. Three examples run against real data: - parse_text_dump_pch: PCH daily snapshot via new_text - parse_text_dump_routeviews: route-views oix snapshot via from_text_reader - parse_text_dump_auto: new_auto on both text dump and MRT file Closes #320
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #321 +/- ##
========================================
Coverage 90.50% 90.51%
========================================
Files 94 95 +1
Lines 20468 21203 +735
========================================
+ Hits 18524 19191 +667
- Misses 1944 2012 +68 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Pull request overview
Adds first-class support for parsing Cisco sh ip bgp fixed-width text RIB dumps (PCH daily snapshots and route-views oix-full-snapshot-*) into BgpElems, and integrates that path into the existing BgpkitParser iteration model so consumers can iterate elements uniformly across MRT and text sources.
Changes:
- Introduces
parser::text_dumpwith detection, header/column inference, fixed-width row parsing, and filename/URL timestamp inference. - Extends
BgpkitParserwithnew_text/from_text_reader*andnew_auto/from_auto_reader*constructors plus iterator integration via a pre-parsed element queue. - Adds examples and updates the changelog to document the new parsing capabilities and unified API.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| src/parser/text_dump.rs | New Cisco fixed-width text dump parser, timestamp inference, and unit tests. |
| src/parser/mod.rs | Adds text-dump element queue to BgpkitParser, new constructors (new_text, new_auto, from_*_reader*), and next_record behavior for text dumps. |
| src/parser/iters/fallible.rs | Drains pre-parsed text-dump elements in FallibleElemIterator with filter support. |
| src/parser/iters/default.rs | Drains pre-parsed text-dump elements in ElemIterator with filter support. |
| examples/parse_text_dump_routeviews.rs | Demonstrates parsing route-views snapshots (with a size cap) via the new text-dump reader API. |
| examples/parse_text_dump_pch.rs | Demonstrates parsing PCH daily snapshot text dumps via new_text. |
| examples/parse_text_dump_auto.rs | Demonstrates new_auto dispatch between MRT and text dump inputs. |
| CHANGELOG.md | Documents the new text dump parser module and unified constructor API. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| pub fn next_record(&mut self) -> Result<MrtRecord, ParserErrorWithBytes> { | ||
| if self.text_dump_elems.is_some() { | ||
| return Err(ParserError::Unsupported( | ||
| "text-dump parsers have no MRT record representation; iterate elements instead" | ||
| .to_string(), |
| let mut buf = Vec::with_capacity(20 * 1024 * 1024); | ||
| let _ = reader.by_ref().take(20 * 1024 * 1024).read_to_end(&mut buf); |
| /// The file is auto-decompressed by oneio. The timestamp for all elements | ||
| /// is inferred from the file name when possible; pass | ||
| /// [`infer_timestamp_from_path`] yourself to override. The resulting parser | ||
| /// iterates over [`BgpElem`]s — calling [`into_record_iter`](Self::into_record_iter) | ||
| /// or [`next_record`](Self::next_record) on a text-dump parser panics, since | ||
| /// text dumps have no MRT-record representation. |
Replace the collect-all parse_text_dump_with_timestamp approach with a streaming TextDumpElemIterator that yields one BgpElem per route line. new_text, from_text_reader, new_auto, and from_auto_reader now store the iterator directly in BgpkitParser instead of materializing a Vec<BgpElem> at construction. ElemIterator and FallibleElemIterator drain it lazily with filter support. This cuts peak memory from O(all elems) to O(1) for text dumps. The full route-views oix snapshot (~900k prefixes, gigabytes of text) now streams without buffering. parse_text_dump / parse_text_dump_with_timestamp are kept as convenience wrappers that collect the iterator. Remove the 20 MB demo cap from the route-views example since streaming makes full-dump processing practical.
Address Copilot review: RecordIterator, FallibleRecordIterator, UpdateIterator, FallibleUpdateIterator, RawRecordIterator, RouteIterator, and FallibleRouteIterator all treated ParserError::Unsupported as a recoverable warning and continued, which caused into_record_iter() and into_update_iter() on text-dump parsers to spin forever. Add an early text_dump_iter.is_some() check to each iterator's next() that returns None immediately. Also fix the new_text doc comment to point to from_text_reader_with_timestamp for timestamp override and clarify the behavior of record-based iterators on text dumps.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 11 out of 11 changed files in this pull request and generated no new comments.
Suppressed comments (4)
src/parser/text_dump.rs:319
as_path_from_tokensonly parses bare u32 tokens, so it will drop valid ASNs formatted with the existingAsnstring syntax (e.g.AS13335). UsingAsn'sFromStrkeeps this parser consistent with other code paths and avoids silently producing empty/incorrect AS paths.
if let Ok(asn) = token.parse::<u32>() {
asns.push(Asn::from(asn));
}
src/parser/text_dump.rs:536
TextDumpElemIterator::nexttreatsread_lineI/O errors as EOF (Err(_) => return None), which can silently truncate parsing (including throughparse_text_dump_with_timestamp, which returnsio::Result). At minimum, surface the failure so callers aren't misled into thinking the dump parsed cleanly.
match self.reader.read_line(&mut self.buf) {
Ok(0) => return None, // EOF
Ok(_) => {}
Err(_) => return None,
}
src/parser/mod.rs:235
- The doc comment says
timestamp"overrides the inferred value", butfrom_auto_reader_with_timestampcannot infer anything (it only has a reader). This is confusing for API consumers; the doc should describetimestampas the value used for text dumps and point tonew_auto(path)for filename-based inference.
/// Create a parser from any reader, auto-detecting MRT vs text dump.
/// When text is detected, `timestamp` overrides the inferred value
/// (`None` → `0.0`).
CHANGELOG.md:39
- PR description states text-dump elements are materialized at construction, but the implementation and this changelog entry describe lazy, streaming parsing via
TextDumpElemIterator. Please reconcile the PR description with the actual behavior so users have accurate performance/memory expectations.
* **Unified MRT/text-dump parser API**: `BgpkitParser` gains three new constructor groups that integrate text dumps into the standard `for elem in parser` iteration loop. `new_text(path)` / `from_text_reader(r)` parse a known text dump; `new_auto(path)` / `from_auto_reader(r)` peek the first bytes and auto-dispatch to the text or MRT path. Both text-dump paths **stream elements lazily** — one route line at a time, constant memory — via a new `TextDumpElemIterator`. All existing filter methods (`add_filter`, `with_filters`, etc.) work on both paths. The default `new(path)` constructor remains MRT-only.
- TextDumpElemIterator: log read_line I/O errors instead of silently
treating them as EOF, so partial reads are visible in logs
- from_auto_reader_with_timestamp: fix doc to clarify that timestamp
sets (not overrides) the text-dump timestamp, since this method has
no path to infer from; point to new_auto for filename-based inference
- as_path_from_tokens: document why bare u32 parsing is correct (Cisco
sh ip bgp output never uses AS{n} notation)
Resolve CHANGELOG conflict: keep both the text-dump entries and the RFC 5543 Traffic Engineering attribute entry from main.
Summary
Ports the Cisco
sh ip bgpfixed-width text dump parser from monocle into bgpkit-parser so it can be used programmatically as a library. Adds unifiedBgpkitParserconstructors that integrate text dumps into the standardfor elem in parseriteration loop, so users no longer need to know the input format in advance.Closes #320.
Changes
New:
src/parser/text_dump.rsFull parser ported from monocle (bgpkit/monocle#143, bgpkit/monocle#146):
detect_text_dump(reader)— sniffs the first 256 bytes to distinguish text dumps from MRT (PCHBGP tablepreamble, route-viewsStatus codes:opening)parse_header(reader)/TextDumpHeader— table version, router ID, local AS, header-derived fixed-width column offsetsparse_text_dump()/parse_text_dump_with_timestamp()→Vec<BgpElem>infer_timestamp_from_path()— PCH (YYYY.MM.DD, noon UTC) and route-views (YYYY-MM-DD-HHMM)0.0.0.0/ AS0)Unified
BgpkitParserAPI (src/parser/mod.rs)Three new constructor groups integrate text dumps into the standard iteration loop:
new_text(path)/from_text_reader(r)— parse a known text dumpnew_auto(path)/from_auto_reader(r)— peek first bytes, auto-dispatch text vs MRTText-dump elements are materialized at construction (RIB snapshots are full snapshots, not streaming). All filter methods (
add_filter,with_filters, etc.) work on both the MRT and text paths. The defaultnew(path)constructor remains MRT-only — zero impact on the existing streaming hot path.next_record()on a text-dump parser returns a clear error (no MRT-record representation).Iterator integration (
src/parser/iters/)ElemIteratorandFallibleElemIteratordrain pre-parsed text-dump elems with filter support, falling through to the MRT record loop when no text elems are present.Test plan
text_dump.rs(detection, header parsing, continuations, wrapped prefixes, origin codes, route-views sentinel behavior, timestamp inference)mod.rs(text reader, auto-detect text, auto-detect MRT fallback, filter on text dump, next_record error)cargo test --all-features)cargo fmt --check✓,cargo clippy --all-targets --all-features -- -D warnings✓new_text(matches monocle count)oix-full-snapshot-2026-07-01-0000(first 20 MB) → 234,333 elements viafrom_text_reader_with_timestampnew_autoon both text dump (97,546 elems) and MRT file (8,160 elems) — auto-dispatch worksExamples
Three new examples under
examples/:parse_text_dump_pch.rs— PCH daily snapshot vianew_textparse_text_dump_routeviews.rs— route-views oix snapshot viafrom_text_reader_with_timestampparse_text_dump_auto.rs—new_autoon both text dump and MRT fileDesign notes
new(path)constructor is untouched (MRT-only, streaming). Text-dump support is opt-in vianew_text/new_auto.