Skip to content

Check the measure ids the US legacy ETL cleanup doc names - #354

Open
MaxGhenis wants to merge 6 commits into
mainfrom
test/legacy-etl-doc-measure-ids
Open

MaxGhenis wants to merge 6 commits into
mainfrom
test/legacy-etl-doc-measure-ids

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Summary

test_us_legacy_etl_cleanup_doc_accounts_for_retired_series (added in #348) checked only that the backticked package ids in docs/us-legacy-etl-cleanup.md are shipped packages. It did not check the measure ids written next to them, so a renamed or mistyped measure id passed CI. An independent review of #350, which edits five of those rows, flagged the gap. Two ids had already drifted on main:

  • The mortgage row named soi-congressional-district-2022 interest_paid_deduction_*. chronicle#345 renamed those measures to mortgage_interest_paid_* (still N19300/A19300; see docs/concept-migrations.md, "Schedule A line 8a relabel").
  • The SSI row put ssi_payments after three packages. It is a row value id of ssa-annual-statistical-supplement-2025's payment_amount measure. ssa-ssi-table-7b1-2024 and ssa-ssi-monthly-statistics-2024-12 declare neither.

Test. The test now parses every "Chronicle package and measures" cell under a grammar stated in a comment above the helpers:

  • A cell splits into clauses at each ; outside parentheses.
  • Within a clause, a backticked id belongs to every package listed since the previous id. So `a`, `b`: `x` claims x for both packages, and `a` `x`, `y` claims both ids for a.
  • An exact id must be a measure id or a record-set row value id that the package declares. A prefix_* wildcard must match at least one of its measure ids.
  • An IRS column code (N07225, A06500, ...) must be a source_column_id read by a measure that the id before it matches.
  • A token that is none of these kinds fails the test, and so does an id with no package before it.

How tables are read. Three adversarial review rounds each found a valid Markdown edit that put a bad claim on the rendered page while the test passed:

  • round 1: an indented row, a row without outer pipes, an escaped \|;
  • round 2: a second table under a bold header, a column named twice;
  • round 3: a table inside a blockquote, list item or <details> block, and an extra table headed Chronicle package and measures\*.

The test now reads the doc with markdown-it-py, which is CommonMark plus GitHub's table extension. A table therefore counts wherever GitHub renders one, under a heading of any form. Every column whose header reads "Chronicle package and measures" once emphasis, code and escape marks are stripped is checked. DOC_TABLES pins every table's heading and header, so an added, moved or reheaded table fails until it is listed. Every reproduction from the three rounds is now a real-doc mutation test. markdown-it-py was already locked at 4.0.0 through rich. This PR declares it as a dev dependency, and the uv.lock diff is metadata only.

The test reads each package's YAML through chronicle.source_package.load_source_package and builds no facts. On main's doc it reports 5 errors: the renamed district measure and its two columns, plus ssi_payments against the two SSI packages that lack it. With this PR's doc it checks 64 claims (52 ids, 12 column codes) and finds none. Packages listed together are each checked: all three ACA OEP packages, both Medicaid packages and the three packages that publish net_capital_gains_*.

New tests:

  • test_us_legacy_etl_cleanup_doc_check_flags_edited_claims applies 10 edits to the real doc. One renames a measure; the rest are the review reproductions. Each must raise exactly the named errors.
  • test_us_legacy_etl_cleanup_doc_check_reads_any_heading_form checks that indented, closed and setext headings raise no error.
  • test_markdown_tables_read_cells_as_github_renders_them and test_replacement_cells_read_every_replacement_column cover the reader.
  • test_replacement_claim_check_flags_undeclared_ids runs one-row fake tables against fake packages. It covers a nonexistent measure id, a wildcard with no match, a wildcard that would match only a row, a listed package that lacks the id, a wrong column code, a column code after a row id, an unknown package, an id or column with nothing before it, an id cut off by ;, and an unreadable token.
  • test_replacement_claim_check_accepts_declared_ids covers the forms the doc uses. It pins the claims each one produces.

Doc. This PR fixes the two rows. It also rewrites the QBI gap note, which still said the broad package's qbi_* read N03270/A03270 "on main". chronicle#345 repinned them, and main's package reads N04475/A04475. The new wording is #350's, character for character, so that line merges cleanly.

Interaction with #350. A trial merge of this branch onto #350's head (a838ee45) conflicts in one hunk: the SALT, mortgage and charitable rows, which #350 rewrites. Taking #350's side resolves it. In the resolved tree, #350's new rows (taxes_paid_deduction_*, broad mortgage_interest_paid_*, charitable_* and taxable_state_local_income_tax_refunds_*, with their column codes) pass the new check, and all 19 tests in this group pass.

Chronicle Governance

This PR does not touch the Chronicle source-data boundary. It adds no source package, target profile, consumer contract or schema, and it changes no fact. It changes a boundary test, a doc, a changelog fragment and the dev dependency list.

  • Approved Chronicle agent role: n/a (no source package, target profile or contract change)
  • Deterministic checks run: uv run ruff check tests/test_chronicle_boundaries.py; uv run pytest tests/test_chronicle_boundaries.py -k "cleanup_doc or replacement_ or markdown_tables" (33 passed)
  • LLM judge verdicts: n/a (no source-data boundary change)

Tests

🤖 Generated with Claude Code

MaxGhenis and others added 6 commits October 12, 2026 22:36
The cleanup doc's test checked only that the backticked package ids in its
"Where the publisher's values live" tables are shipped packages. A renamed
or mistyped measure id next to one passed CI, and two had drifted:

- soi-congressional-district-2022 still named interest_paid_deduction_*,
  which chronicle#345 renamed to mortgage_interest_paid_* (N19300/A19300).
- The SSI row put ssi_payments after three packages; it is a row of
  ssa-annual-statistical-supplement-2025's payment_amount, and the other
  two packages declare neither.

The test now parses each "Chronicle package and measures" cell: an id
belongs to every package listed since the previous id in its clause, must
be a measure id or row value id that package declares (a prefix_* wildcard
must match a measure), and an IRS column code must be a source column a
matching measure reads. Unit tests cover the grammar with fake packages,
including a nonexistent measure id, and a mutation of the real doc. It
reads package YAML only and builds no facts.

The doc fixes those two rows and the stale QBI note that the broad
package's qbi_* read N03270/A03270 "on main" (chronicle#345 repinned them),
worded as chronicle#350 words it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ed package

The renamed-measure mutation test reuses the declarations the doc test
loaded, and the cell check looks for a backticked package id rather than
any hyphenated word.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review of #354 found that valid Markdown slipped past the claim check.
An indented row, or a row without outer pipes, ended the table early, so
that row and the ones after it were never read. An escaped pipe split a
cell, which dropped a wrong column code.

A table now runs from its header to the next blank line. Rows may be
indented or drop their outer pipes, and `\|` stays inside its cell. A row
whose cell count differs from the header's fails. The doc test also
requires one table per "Where the publisher's values live" section. The
real-doc mutation test now covers each bypass the review reproduced; the
old reader reported no error for any of them.

The package-path scan is cached, and the grammar comment says that
historical ids go without backticks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The delta review of #354 found two more silent bypasses.
- A repeated header written as **Chronicle package and measures** after a
  blank line started a second table, and the reader skipped its rows.
- A header that named the column twice had only its first copy checked.

Tables are now found as GitHub Markdown finds them: a row followed by a
delimiter row with the same number of cells. Every column whose header,
read without emphasis or code marks, is "Chronicle package and measures"
is checked. Each "Where the publisher's values live" section must also
hold exactly one table with the standard header. Both reproductions are
real-doc mutation cases, and a mutation can now apply several edits.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The third review of #354 found that a replacement table inside a
blockquote, list item or <details> block was skipped. It also found that
valid heading forms (indented, closing hashes, setext) raised a false
layout error, and that an extra table headed with an escaped asterisk
went unread.

The test now reads the doc with markdown-it-py (CommonMark plus GitHub's
table extension), so a table counts wherever GitHub renders one, under a
heading of any form. Cells are read as GitHub reads them: `\|` is a pipe,
short rows are padded and extra cells are dropped. DOC_TABLES pins every
table's heading and header. A table added, moved or reheaded fails until
it is listed, and the replacement-column check still reads it in the
meantime. markdown-it-py was already locked at 4.0.0 through rich; it is
now a declared dev dependency, and the lock gains metadata only.

Every reproduction from the three reviews is a real-doc mutation case,
and reformatted headings are a passing case.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis

Copy link
Copy Markdown
Contributor Author

Independent review trail. Each round was GPT-6.1 Sol, adversarial, run through Subfleet.

Round Head Verdict Blocking findings Fixed in
1 c0f750f REQUEST_CHANGES Valid Markdown skipped or split rows: an indented row, a row without outer pipes, an escaped | 44f43f3
2 44f43f3 REQUEST_CHANGES A second table under a bold header was skipped, and a repeated column was half-checked 309dc8a
3 309dc8a REQUEST_CHANGES A table inside a blockquote, list item or <details> block was skipped ebffab4 (markdown-it-py reader, DOC_TABLES pin)
4 ebffab4 APPROVE none —

Every reproduction from rounds 1-3 is now a case in DOC_MUTATIONS.

Round 4 left one nit: a prose line placed directly under a table, with no blank line between, fails the doc test. That is the correct result, not a false failure. GitHub's own GFM spec (cmark-gfm test/spec.txt, the example after "The table is broken at the first empty line") absorbs such a line as an extra table row. GitHub would therefore render it as a row with an empty "Chronicle package and measures" cell, and markdown-it-py reads it the same way. The cmark-gfm extensions.txt fixture cited in that round has a blank line before Hi!, which is why it ends the table there.

Round 4's sandbox could not run the lock check. I ran uv lock --check locally and it passes; the uv.lock diff is metadata only.

Merge note for chronicle#350: whichever PR merges second has one conflicting hunk in docs/us-legacy-etl-cleanup.md, the SALT, mortgage and charitable rows. Take #350's version of those rows. The reader returns the same 30 cells and the same table layout on #350's doc, and #350's new rows pass the check.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant