Skip to content

feat: add Cisco sh ip bgp text dump parser with unified MRT/text API - #321

Merged
digizeph merged 5 commits into
mainfrom
feat/text-dump-parser
Aug 6, 2026
Merged

feat: add Cisco sh ip bgp text dump parser with unified MRT/text API#321
digizeph merged 5 commits into
mainfrom
feat/text-dump-parser

Conversation

@digizeph

@digizeph digizeph commented Aug 5, 2026

Copy link
Copy Markdown
Member

Summary

Ports the Cisco sh ip bgp fixed-width text dump parser from monocle into bgpkit-parser so it can be used programmatically as a library. Adds unified BgpkitParser constructors that integrate text dumps into the standard for elem in parser iteration loop, so users no longer need to know the input format in advance.

Closes #320.

Changes

New: src/parser/text_dump.rs

Full parser ported from monocle (bgpkit/monocle#143, bgpkit/monocle#146):

  • detect_text_dump(reader) — sniffs the first 256 bytes to distinguish text dumps from MRT (PCH BGP table preamble, route-views Status codes: opening)
  • parse_header(reader) / TextDumpHeader — table version, router ID, local AS, header-derived fixed-width column offsets
  • parse_text_dump() / parse_text_dump_with_timestamp()Vec<BgpElem>
  • infer_timestamp_from_path() — PCH (YYYY.MM.DD, noon UTC) and route-views (YYYY-MM-DD-HHMM)
  • Multipath continuations, wrapped prefixes (including IPv6), AS-set flattening — all preserved
  • Route-views dumps without preamble → sentinel peer identity (0.0.0.0 / AS0)

Unified BgpkitParser API (src/parser/mod.rs)

Three new constructor groups integrate text dumps into the standard iteration loop:

  • new_text(path) / from_text_reader(r) — parse a known text dump
  • new_auto(path) / from_auto_reader(r) — peek first bytes, auto-dispatch text vs MRT

Text-dump elements are materialized at construction (RIB snapshots are full snapshots, not streaming). All filter methods (add_filter, with_filters, etc.) work on both the MRT and text paths. The default new(path) constructor remains MRT-only — zero impact on the existing streaming hot path.

next_record() on a text-dump parser returns a clear error (no MRT-record representation).

Iterator integration (src/parser/iters/)

ElemIterator and FallibleElemIterator drain pre-parsed text-dump elems with filter support, falling through to the MRT record loop when no text elems are present.

Test plan

  • 22 unit tests in text_dump.rs (detection, header parsing, continuations, wrapped prefixes, origin codes, route-views sentinel behavior, timestamp inference)
  • 5 constructor integration tests in mod.rs (text reader, auto-detect text, auto-detect MRT fallback, filter on text dump, next_record error)
  • 809 tests pass, 0 failed (cargo test --all-features)
  • cargo fmt --check ✓, cargo clippy --all-targets --all-features -- -D warnings
  • Real-data verification:
    • PCH bom2 2026-07-01 dump → 97,546 elements via new_text (matches monocle count)
    • route-views oix-full-snapshot-2026-07-01-0000 (first 20 MB) → 234,333 elements via from_text_reader_with_timestamp
    • new_auto on both text dump (97,546 elems) and MRT file (8,160 elems) — auto-dispatch works

Examples

Three new examples under examples/:

  • parse_text_dump_pch.rs — PCH daily snapshot via new_text
  • parse_text_dump_routeviews.rs — route-views oix snapshot via from_text_reader_with_timestamp
  • parse_text_dump_auto.rsnew_auto on both text dump and MRT file

Design notes

  • The default new(path) constructor is untouched (MRT-only, streaming). Text-dump support is opt-in via new_text / new_auto.
  • Text dumps are full snapshots — the parser materializes all elements at construction. For a ~900k-prefix global snapshot this is a one-time allocation, after which iteration is zero-cost.

Port the fixed-width text RIB dump parser from monocle (#143, #146) into
bgpkit-parser so it can be used programmatically. The new parser::text_dump
module handles PCH daily snapshots and route-views oix-full-snapshot files,
extracting column offsets from the table header, handling multipath
continuations and wrapped prefixes (including IPv6), and tolerating missing
preambles (route-views sentinel peer identity).

Add unified BgpkitParser constructors so text dumps integrate into the
standard for-elem-in-parser loop:

- new_text(path) / from_text_reader(r): parse a known text dump
- new_auto(path) / from_auto_reader(r): peek first bytes, auto-dispatch
  text vs MRT

Text-dump elements are materialized at construction (RIB snapshots are full
snapshots, not streaming). All filter methods work on both paths. The default
new(path) constructor remains MRT-only.

22 unit tests + 5 constructor integration tests cover detection, header
parsing, continuations, wrapped prefixes, origin codes, route-views sentinel
behavior, timestamp inference, auto-detection, and filter integration.

Three examples run against real data:
- parse_text_dump_pch: PCH daily snapshot via new_text
- parse_text_dump_routeviews: route-views oix snapshot via from_text_reader
- parse_text_dump_auto: new_auto on both text dump and MRT file

Closes #320
Copilot AI review requested due to automatic review settings August 5, 2026 20:55
@codecov

codecov Bot commented Aug 5, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 89.52381% with 77 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.51%. Comparing base (d225c91) to head (8596c3a).

Files with missing lines Patch % Lines
src/parser/text_dump.rs 91.38% 48 Missing ⚠️
src/parser/mod.rs 87.41% 18 Missing ⚠️
src/parser/iters/fallible.rs 40.00% 6 Missing ⚠️
src/parser/iters/route.rs 66.66% 2 Missing ⚠️
src/parser/iters/update.rs 66.66% 2 Missing ⚠️
src/parser/iters/raw.rs 66.66% 1 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff            @@
##             main     #321    +/-   ##
========================================
  Coverage   90.50%   90.51%            
========================================
  Files          94       95     +1     
  Lines       20468    21203   +735     
========================================
+ Hits        18524    19191   +667     
- Misses       1944     2012    +68     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds first-class support for parsing Cisco sh ip bgp fixed-width text RIB dumps (PCH daily snapshots and route-views oix-full-snapshot-*) into BgpElems, and integrates that path into the existing BgpkitParser iteration model so consumers can iterate elements uniformly across MRT and text sources.

Changes:

  • Introduces parser::text_dump with detection, header/column inference, fixed-width row parsing, and filename/URL timestamp inference.
  • Extends BgpkitParser with new_text / from_text_reader* and new_auto / from_auto_reader* constructors plus iterator integration via a pre-parsed element queue.
  • Adds examples and updates the changelog to document the new parsing capabilities and unified API.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
src/parser/text_dump.rs New Cisco fixed-width text dump parser, timestamp inference, and unit tests.
src/parser/mod.rs Adds text-dump element queue to BgpkitParser, new constructors (new_text, new_auto, from_*_reader*), and next_record behavior for text dumps.
src/parser/iters/fallible.rs Drains pre-parsed text-dump elements in FallibleElemIterator with filter support.
src/parser/iters/default.rs Drains pre-parsed text-dump elements in ElemIterator with filter support.
examples/parse_text_dump_routeviews.rs Demonstrates parsing route-views snapshots (with a size cap) via the new text-dump reader API.
examples/parse_text_dump_pch.rs Demonstrates parsing PCH daily snapshot text dumps via new_text.
examples/parse_text_dump_auto.rs Demonstrates new_auto dispatch between MRT and text dump inputs.
CHANGELOG.md Documents the new text dump parser module and unified constructor API.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/parser/mod.rs
Comment on lines 179 to +183
pub fn next_record(&mut self) -> Result<MrtRecord, ParserErrorWithBytes> {
if self.text_dump_elems.is_some() {
return Err(ParserError::Unsupported(
"text-dump parsers have no MRT record representation; iterate elements instead"
.to_string(),
Comment thread examples/parse_text_dump_routeviews.rs Outdated
Comment on lines +24 to +25
let mut buf = Vec::with_capacity(20 * 1024 * 1024);
let _ = reader.by_ref().take(20 * 1024 * 1024).read_to_end(&mut buf);
Comment thread src/parser/mod.rs Outdated
Comment on lines +105 to +110
/// The file is auto-decompressed by oneio. The timestamp for all elements
/// is inferred from the file name when possible; pass
/// [`infer_timestamp_from_path`] yourself to override. The resulting parser
/// iterates over [`BgpElem`]s — calling [`into_record_iter`](Self::into_record_iter)
/// or [`next_record`](Self::next_record) on a text-dump parser panics, since
/// text dumps have no MRT-record representation.
Replace the collect-all parse_text_dump_with_timestamp approach with a
streaming TextDumpElemIterator that yields one BgpElem per route line.
new_text, from_text_reader, new_auto, and from_auto_reader now store the
iterator directly in BgpkitParser instead of materializing a Vec<BgpElem> at
construction. ElemIterator and FallibleElemIterator drain it lazily with
filter support.

This cuts peak memory from O(all elems) to O(1) for text dumps. The full
route-views oix snapshot (~900k prefixes, gigabytes of text) now streams
without buffering. parse_text_dump / parse_text_dump_with_timestamp are kept
as convenience wrappers that collect the iterator.

Remove the 20 MB demo cap from the route-views example since streaming makes
full-dump processing practical.
Address Copilot review: RecordIterator, FallibleRecordIterator,
UpdateIterator, FallibleUpdateIterator, RawRecordIterator, RouteIterator,
and FallibleRouteIterator all treated ParserError::Unsupported as a
recoverable warning and continued, which caused into_record_iter() and
into_update_iter() on text-dump parsers to spin forever.

Add an early text_dump_iter.is_some() check to each iterator's next() that
returns None immediately. Also fix the new_text doc comment to point to
from_text_reader_with_timestamp for timestamp override and clarify the
behavior of record-based iterators on text dumps.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated no new comments.

Suppressed comments (4)

src/parser/text_dump.rs:319

  • as_path_from_tokens only parses bare u32 tokens, so it will drop valid ASNs formatted with the existing Asn string syntax (e.g. AS13335). Using Asn's FromStr keeps this parser consistent with other code paths and avoids silently producing empty/incorrect AS paths.
        if let Ok(asn) = token.parse::<u32>() {
            asns.push(Asn::from(asn));
        }

src/parser/text_dump.rs:536

  • TextDumpElemIterator::next treats read_line I/O errors as EOF (Err(_) => return None), which can silently truncate parsing (including through parse_text_dump_with_timestamp, which returns io::Result). At minimum, surface the failure so callers aren't misled into thinking the dump parsed cleanly.
            match self.reader.read_line(&mut self.buf) {
                Ok(0) => return None, // EOF
                Ok(_) => {}
                Err(_) => return None,
            }

src/parser/mod.rs:235

  • The doc comment says timestamp "overrides the inferred value", but from_auto_reader_with_timestamp cannot infer anything (it only has a reader). This is confusing for API consumers; the doc should describe timestamp as the value used for text dumps and point to new_auto(path) for filename-based inference.
    /// Create a parser from any reader, auto-detecting MRT vs text dump.
    /// When text is detected, `timestamp` overrides the inferred value
    /// (`None` → `0.0`).

CHANGELOG.md:39

  • PR description states text-dump elements are materialized at construction, but the implementation and this changelog entry describe lazy, streaming parsing via TextDumpElemIterator. Please reconcile the PR description with the actual behavior so users have accurate performance/memory expectations.
* **Unified MRT/text-dump parser API**: `BgpkitParser` gains three new constructor groups that integrate text dumps into the standard `for elem in parser` iteration loop. `new_text(path)` / `from_text_reader(r)` parse a known text dump; `new_auto(path)` / `from_auto_reader(r)` peek the first bytes and auto-dispatch to the text or MRT path. Both text-dump paths **stream elements lazily** — one route line at a time, constant memory — via a new `TextDumpElemIterator`. All existing filter methods (`add_filter`, `with_filters`, etc.) work on both paths. The default `new(path)` constructor remains MRT-only.

- TextDumpElemIterator: log read_line I/O errors instead of silently
  treating them as EOF, so partial reads are visible in logs
- from_auto_reader_with_timestamp: fix doc to clarify that timestamp
  sets (not overrides) the text-dump timestamp, since this method has
  no path to infer from; point to new_auto for filename-based inference
- as_path_from_tokens: document why bare u32 parsing is correct (Cisco
  sh ip bgp output never uses AS{n} notation)
Resolve CHANGELOG conflict: keep both the text-dump entries and the
RFC 5543 Traffic Engineering attribute entry from main.
@digizeph
digizeph enabled auto-merge August 6, 2026 23:29
@digizeph
digizeph merged commit 3f2b0cb into main Aug 6, 2026
8 checks passed
@digizeph
digizeph deleted the feat/text-dump-parser branch August 6, 2026 23:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Port Cisco/route-views sh ip bgp text dump parser from monocle

2 participants