Skip to content

fix(vendor): render both IANA citation spellings as :rfc: roles (#1664) - #1673

Merged
JarryShaw merged 2 commits into
mainfrom
fix/1664-vendor-rfc-roles
Oct 13, 2026
Merged

JarryShaw merged 2 commits into
mainfrom
fix/1664-vendor-rfc-roles

Conversation

@JarryShaw

@JarryShaw JarryShaw commented Oct 13, 2026 •

Copy link
Copy Markdown
Owner

Please follow the guide below

  • You will be asked some questions, please read them carefully and answer honestly

  • Put an x into all the boxes [ ] relevant to your pull request (like that [x])

  • Use Preview tab to see how your pull request will actually look like

  • Searched for similar pull requests

  • Followed the coding style (make pylint, make mypy, make isort): nothing new on the changed crawlers, and test_isort_clean passes

  • make test passes, and a test case covers the change: a new test covers it, but only tests/vendor, tests/const, test_isort_clean and test_docstring_contract were run

  • Added a changelog entry under docs/source/changelog/ and regenerated CHANGELOG.md, if the change is user-visible: N/A — centralised in docs(changelog): shared 1.5.0 changelog — long-lived, merges last (#610, #616, #617, #618, #620) #657

What is the purpose of your pull request?

Tick the commit type your subject line carries.

  • fix — corrects a defect
  • feat — adds a feature
  • perf — changes performance, not behaviour
  • refactor — changes neither behaviour nor performance
  • test — tests only
  • docs — documentation only
  • ci — workflows or build tooling
  • release — bumps the version or rolls up a distribution
  • chore — anything else

Description of your pull request and other information

Root cause: IANA, not our code. IANA re-exported its CSVs on 2026-10-13 (Last-Modified 17:26–17:27Z, then 17:52–17:54Z). The XML records behind them still read <xref type="rfc" data="rfc1072"/>. TCP option kind 6, Wayback 2025-11-21 against live:

6,6,Echo (obsoleted by option 8),[RFC1072][RFC6247]
6,6,Echo (obsoleted by option 8),[RFC 1072]; [RFC 6247]

The crawlers recognised only RFC written against its number, and they kept ; as a citation. Each run fixed a different set of files: the second wave was exported after run 1 (17:47Z), and Cloudflare (s-maxage=1800) can serve a copy up to 30 minutes old (that part is inferred). The 1.5.0b12 bump (2025da5) then broke the last 18 HIP/MH files, and the second commit regenerates them with the fixed crawlers, matching b12's parent 2575804 byte for byte. The port registry switched spelling first, and on 2026-09-29 the 1.5.0b6 bump (59dc8c0) dropped the roles from 670 AppType lines.

Fix. The new Vendor.split_references() drops ;/whitespace separator segments and writes RFC 1072 as RFC1072. All 64 reference loops now call it. The ESP and TLS key label regexes accept both spellings, and FTP renders [RFC 6384, Section 11] as :rfc:`6384#section-11` .

Regeneration (pcapkit-vendor then isort, as cron-vendor.yml runs them): 67 of the 73 files match c3380134d byte for byte. AppType ×5 regains its pre-b6 roles. Two rows change because the exporter changed, not the data: robustirc's description now reads ([RFC 1459]), and FTP ALGS was formerly [RFC6384] Section 11. Every const file's AST, docstrings stripped, equals c3380134d's, and once roles are ignored those two rows are the only text that differs. So no registry content changed.

Bump guard (not here; workflows are out of scope). cron-vendor.yml bumps whenever verify-changed-files finds any change, so a comment-only regeneration still bumps. Comparing each changed const file's AST, with docstrings stripped, against HEAD classifies b9, b10 and b11 as comment-only and still bumps on b8's member rename. b9 was a real draft-to-RFC 10049 update, so whether comment-only changes get committed without a bump, or not at all, is for the owner to decide.

Tests. The new test_vendor_reference_formats_1664_unit.py runs old and new cells through each crawler variant. Its 11 tests give FAILED (failures=8, errors=6) on origin/main and OK here. The PR also fixes test_vendor_ipv6_extension_header_925_unit's byte-for-byte test, which currently fails on origin/main.

Closes #1664

@JarryShaw JarryShaw added fix Pull requests that fix a defect (fix: subject prefix) const Regenerated IANA or vendor constant tables; members keep their numeric values review: good-to-go Cross-review at the current head says ready; CI state is separate labels Oct 13, 2026
@JarryShaw

Copy link
Copy Markdown
Owner Author

Verdict: GOOD TO GO at 9303de84e. A Sonnet cross-review re-derived every load-bearing claim and found no defect. I re-ran the new test, tests/vendor and tests/const: 11, 129 and 330 passed.

Root cause: IANA's CSV export, not our code. IANA re-exported every registry CSV today in two waves, at 17:26–17:27Z and 17:52–17:53Z. Citations moved from [RFC1072][RFC6247] to [RFC 1072]; [RFC 6247]; the XML is unchanged. The crawlers matched only the unspaced form. b10 and b11 rewrote different files because the CDN caches IANA for 30 minutes (inferred). 18 hip/mh files on main still hold the old form, so the next Vendor Update would bump b12 without this.

Fix. Vendor.split_references() (pcapkit/vendor/default.py) reads both spellings and is used by all 64 reference loops. 67 of the 73 const files the bot touched come back byte-identical to c3380134d, and no other const file changed. An AST comparison finds no code change at all.

For you to accept: the five reg/apptype/* files restore about 670 :rfc: roles. Those were already lost in 1.5.0b6, when the port registry switched to spaced citations, so this goes beyond c3380134d.

Not in scope, and needs a workflow change: Vendor Update still bumps on a docstring-only regeneration (tj-actions/verify-changed-files). A guard that compares each changed module's AST without docstrings would stop that.

@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Coverage: 89.46% (unit tier, Python 3.14, bd5263243, Unit Tests run success)

Package Statements Missed Branches Partial Cover
pcapkit (top level) 104 3 20 3 95.16%
pcapkit/const 18858 954 2366 868 91.06%
pcapkit/corekit 2366 67 760 35 96.48%
pcapkit/dumpkit 275 3 100 3 98.40%
pcapkit/foundation 3031 152 1074 35 94.23%
pcapkit/interface 138 7 52 5 93.68%
pcapkit/protocols 17166 226 4580 190 98.07%
pcapkit/toolkit 1350 275 558 6 77.20%
pcapkit/utilities 494 4 146 4 98.75%
pcapkit/vendor 4417 2342 1010 157 43.38%

Per-file detail: the coverage-html artifact of this run.

@JarryShaw

Copy link
Copy Markdown
Owner Author

Hold this one for a round. The unfixed bot pushed 2025da5e5 (1.5.0b12) at about 19:57Z. It rewrote the 18 hip/mh const files that were still in the old format into the broken [RFC 7401] form.

This PR does not touch those files, so it still merges cleanly, but the merge would keep b12's broken copies. A worker is rebasing onto main and regenerating them with the fixed crawler, checking against 257580444. The new head then gets a Sonnet check. This head stays good as far as it goes.

The b12 Create Release run ships those 18 broken files, so you may want to reject it.

IANA's CSV exports now spell a reference cell `[RFC 1072]; [RFC 6247]`
where they wrote `[RFC1072][RFC6247]`. The crawlers matched only the old
spelling, so Vendor Update's 1.5.0b10 and 1.5.0b11 bumps rewrote 73
const files and dropped 1147 of the 1432 `:rfc:` roles under
pcapkit/const, leaving lists such as `[RFC 1072][; ][RFC 6247]`.

- New `Vendor.split_references()` drops `;`/whitespace-only separator
  segments and spells `RFC 1072` as `RFC1072`; all 64 reference loops use it.
- ESP cipher/integrity and TLS key label accept `[RFC 9395]`; FTP renders
  `[RFC 6384, Section 11]` as `:rfc:`6384#section-11``.
- Regenerate the 73 files with `pcapkit-vendor`: 67 match c338013 byte
  for byte; AppType regains the roles 1.5.0b6 dropped from 670 lines when
  the port registry switched spelling first.

New test_vendor_reference_formats_1664_unit.py feeds old and new cells
through each crawler variant; 11 tests fail on origin/main, pass here.
Vendor Update's 1.5.0b12 bump (2025da5) ran the unfixed crawlers over
the 18 hip/mh registries this PR had not yet regenerated, leaving their
references as `[RFC 7401][; ][RFC 6247]` with no `:rfc:` roles.

- Regenerate those 18 files with the fixed crawlers (`python -m
  pcapkit.vendor hip.cipher ... mh.traffic_selector`, then
  `isort -l100 -ppcapkit`) from IANA's live CSVs: all 18 match 2575804,
  b12's parent, byte for byte, restoring 243 roles.

tests/vendor (129 passed) and tests/const (330 passed) pass.
@JarryShaw
JarryShaw force-pushed the fix/1664-vendor-rfc-roles branch from 9303de8 to bd52632 Compare October 13, 2026 20:11
@JarryShaw

Copy link
Copy Markdown
Owner Author

Verdict: GOOD TO GO at bd5263243. It is rebased onto b12 (2025da5e5), and a Sonnet check confirmed the result.

The new commit regenerates the 18 hip/mh files b12 broke, using the fixed crawler. All 18 are now byte-identical to 257580444, b12's parent; I checked the blob hashes myself. Their :rfc: count goes 243 → 0 (b12) → 243.

The reviewer then ran its own regeneration at about 20:08Z, after a third IANA export at 19:53Z that uses the same spelling. All 91 const files this PR changes came out byte-identical to the branch.

  • [; ] and unspaced [RFC<n> citations: 0 left.
  • Remaining [RFC <n>] text: 3 hits, none of them a reference column (IANA's own description text, plus one hand-written comment).

The version files equal main's. Tests: vendor 129, const 330, test_isort_clean and test_docstring_contract all pass.

Once this merges, the next Vendor Update should find nothing to change, so it bumps nothing.

@JarryShaw
JarryShaw merged commit fc69bb1 into main Oct 13, 2026
46 of 47 checks passed
@github-project-automation github-project-automation Bot moved this from In review to Done in PyPCAPKit Oct 13, 2026
@JarryShaw
JarryShaw deleted the fix/1664-vendor-rfc-roles branch October 13, 2026 20:52
@JarryShaw JarryShaw removed the review: good-to-go Cross-review at the current head says ready; CI state is separate label Oct 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

const Regenerated IANA or vendor constant tables; members keep their numeric values fix Pull requests that fix a defect (fix: subject prefix)

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

fix(vendor): Vendor Update drops every :rfc: role from the const files and bumps a beta each run

1 participant