Repository navigation
Aggregate-table captures and readers, part 1 (D4) - #574
Merged
Merged
Conversation
Plain-request captures of the SSA Supplement tables, the POMS SI 01415 annual payment tables and coverage sections, Federal Reserve Z.1, DOL private-pension and IRS IRA tables, with per-source manifests (time, size, SHA-256, URL) and additive registries of 861 numeric tables and 13 references. A folder-level .gitattributes stores the bytes as fetched. Built by a Subfleet build lane in a standalone clone; committed by the coordinating session. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Typed cells and metadata, the registry loader, and parsers for the SSA, POMS, coverage, Z.1, DOL and IRS sources. Every reader checks the capture's size and SHA-256 before reading a cell; missing or suppressed cells stay missing, and published discrepancies keep flags rather than repairs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Invented-input properties for the number, HTML-span, date and workbook handling, and checks of the real captures against independent raw-cell reads and published identities. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
The POMS reader kept only the last Code/Definition table it saw, so
California 2004 lost codes A-F. It now reads every definition table in
the State's section, merges them, and refuses a code that two tables
define differently.
Fixing that surfaced three more reader problems, now fixed:
- The 2009 and 2010 headings "New Jersey (NJ)" were not recognized, so
four New Jersey charts were labelled Nevada and given both States'
definitions. The registry entries are renamed to new_jersey.
- Iowa 1999-2005 defines codes jointly ("C/H", "D/I"). A joint row now
defines each code it names, and Vermont's chart code "A or B"
resolves to both codes.
- A full-width note row in the Massachusetts 1999 definitions was read
as a code. It is now skipped.
Fifteen Iowa and Pennsylvania charts publish codes that no definition
table in their section defines. They are recorded in
poms_open_questions.json, and no definition is inferred for them.
New tests: every chart code resolves or is recorded; definition codes
match an independent raw-HTML locator on all 777 charts; registry
selectors equal the structural reader; each chart's State matches its
date sentence; a California 2004 regression; and Hypothesis properties
for definitions split across tables and for joint codes. Each of the
four bugs, reintroduced, fails at least two of them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The aggregate-table readers are opt-in source readers that nothing in the historical projection imports. List their five source files as post-review exclusions in the birth-evidence script and its expected tuple, and assert that none of the modules is reachable from the birth-evidence reducer. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Definitions are collected per State section heading. A State that is named only in an "X Effective" line has no heading, so the walker would have kept the previous State's definition tables. It now refuses. No pinned capture changes State that way; the regenerated registry is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
describe_poms_tables gave each chart the definition tables seen so far, so a Code/Definition table between two charts of one State section reached only the later chart. The source does not say which earlier chart such a table serves, so the reader now refuses it. No pinned capture has this order: all 777 charts describe as before. Tests: - an invented regression of the reviewer's ordering; - a Hypothesis property over any order of definition tables and charts in a section: a table after a chart refuses, and otherwise every chart reads every definition table of its section; - the raw-HTML differential now locates State sections and definition tables itself, and checks them against both the registry and the live reader. Its joint-code split no longer uses the production helper. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The shared HTML tree never closes a paragraph implicitly, so an unclosed "<p>Nevada" held the heading and chart that followed it. Its whole-subtree text missed the State heading match, and California's selector and definitions carried into the Nevada chart. The State, effective-date and claimant checks now read a paragraph or heading's own inline text, up to the first nested element whose start tag closes an open paragraph (checked against html5lib 1.1; table only outside quirks mode). Nested blocks are still walked in turn. With no nested block this equals normalized_text, and none of the 68,336 paragraphs and headings in the 28 pinned SI 01415 captures nests one: all 777 registered selectors, definitions and parsed cells are unchanged, by hash. Tests, on invented HTML only: the reviewer's minimized regression in both forms, a property that selectors do not depend on omitted </p> ends, and a property that a block with no nested block reads its full text. With the old reader the regression and the omitted-end property fail. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The fourth review found four problems: - e70848e cut a heading's text at blocks that HTML keeps inside a heading, so <h3><div>Nevada</div></h3> again kept California's definitions; - an omitted </p> in a stacked payment cell merged its amounts; - a page of thousands of unclosed paragraphs exhausted the stack; - the closer set lacked dialog and search. The shared HTML tree now follows the WHATWG "in body" rules for optional end tags: - it ends a paragraph, in button scope, at every start tag that closes one, with table read as in standards mode; - it ends a heading when another heading starts while that heading is the current node; - a new cell or row closes the open one, along with anything left open inside it. The POMS reader now: - reads a heading's text through nested divs, sections and paragraphs, and stops only at another heading or a table; - reads a div as a block that can carry a State heading; - refuses a claimant line that names a State other than its section's; - walks the document without recursion. Nothing changes on the pinned captures. The 43 HTML captures build identical trees, and all 874 registered entries hash identically: their cells, the 777 POMS selectors and the POMS definitions. Against html5lib 1.1, the new tree agrees on 20,000 random pages of paragraphs, headings and block containers with optional </p>; the old tree disagreed on 6,639. The remaining differences are of two kinds. Inline end tags misnested across an open paragraph are a documented limit. dialog and search postdate html5lib 1.1. Tests, on invented HTML only: - blocks inside a heading, and a div as a heading; - every paragraph closer, from a list copied from the specification; - an effective date and a State date line inside a heading; - a heading left open before its table; - a property that wrapping heading text in blocks keeps the selectors; - the claimant-State refusal; - deep pages; - stacked cells and table grids with optional end tags; - a property on the tree's structure. Removing each of eleven rules in turn fails at least one test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The fifth review found one medium and three low problems: - 401b3bc read a heading's text through nested blocks up to the next table. A heading that holds a block after its words, such as <h3>Nevada<p>Invented note</p>, then no longer matched the State pattern, and the reader kept the previous State's selector. Claimant headings failed the same way. The tree also ended a heading only at its own end tag, where HTML ends it at any h1-h6 end tag. - thead, tbody, tfoot, caption and colgroup start tags did not end an open cell or row, so a header cell took in the next section's text. This was already in the parent. - _own_text still recursed: a heading nesting 1,500 elements raised RecursionError. - the claimant-State guard refused safe lines: "Washington, D.C." in the District's section, and a claimant line before any State heading. The shared HTML tree now: - ends the open heading in scope at any h1-h6 end tag; - clears the stack back to the table at a caption, colgroup, thead, tbody or tfoot start tag, and ends an open caption at a tr, td or th start tag. col is left out, since HTML places it in an open colgroup, as all 4,045 col tags in the 28 pinned captures that have one are placed. The POMS reader now: - reads a heading as lines split at its nested blocks, and checks the whole text first, then each line, for the State, the claimant and the effective date. A whole-text "State Effective ..." line stays a date line; - refuses a heading whose lines name two States; - reads "Washington, D.C." as the District of Columbia; - skips the claimant-State guard until the first State heading, since a chart there refuses anyway; - collects text without recursion. The cell reader, _cell_token, recursed too: a cell nesting 1,500 spans raised RecursionError. It now keeps its own stack. Invariants: - A State or claimant heading names its State or claimant whatever blocks follow its words inside it, and whichever heading end tag closes it. - Selectors do not depend on omitted end tags of paragraphs, headings, cells, rows, table sections or captions. - No reader's recursion depth grows with the page's nesting. - The iterative cell reader gives the recursive reader's text, flags and header identity. Nothing changes on the pinned captures. The tree comparison builds 44 identical trees, and all 874 registered entries hash identically. Round 4's message called the compared files "43 HTML captures": the registry holds 41 HTML captures, and the comparison's byte filter also admits two xlsx workbooks and a zip archive. Against html5lib 1.1, on 5,000 invented pages per mode: - headings left open, holding blocks or closed by another level: 0 mismatches (401b3bc: 506); - tables with optional cell, row, section and caption ends: 0 (401b3bc: 1,230); - round 4's paragraph generator: 59 for both trees, from span and b misnesting and from dialog and search, which the tree does not model. Tests, on invented HTML only: - the reviewer's heading cases: blocks after a State or claimant heading's words, open headings and mismatched end tags; - an effective line after heading words, a table inside an open heading, and a heading whose lines name two States; - deep headings, paragraphs and cells; - a claimant line before any State heading, and Washington, D.C.; - table sections, captions and colgroups ending open cells and rows; - heading end tags ending any heading in scope, and stopping at scope; - a property that wrapping or ending heading text keeps the selectors; - a differential property of the cell reader against the recursive one. 368 tests pass. Each of 27 mutants of the tree and the POMS reader, including two the review noted survived round 4's tests, fails at least one test, as does each of five mutants of the cell reader. Documented limits: end tags other than headings close at the matching element regardless of scope, and a br or nested block in one cell joins its lines with a space. No pinned cell, or paragraph of a cell, has a number on each side of a br. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Master gained D1a (#575). Both branches extended the birth-evidence check's post-review exclusions and its reachability test, so the merge keeps D1a's two PSID income and wealth readers and D4's five aggregate-table modules, each with its own reachability assertion. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Medium: every line of a block now passes both State guards. A line that dates another State refuses, and the claimant guard runs on every claimant line, while the claimant still comes from the first. Every block that ends a paragraph, apart from a table or rule, is read from its own text, so a list item, section or blockquote reads the same after a closed heading as inside one left open. A heading whose first line names a State other than its section's refuses. Low 2: _payment_rows no longer deep-copies the chart. It builds a skeleton of the table without recursion, whose cells hold only a key to the original cell. A chart cell nesting 50,000 spans now reads in its own process; at 697ea34 that process exits 139. As before, a table nested in a cell is that cell's text, and its rows are not the chart's. Low 3: a line also ends at a br, and lines drop U+200B and U+FEFF. A postal code after a State's name must be that State's own, so "New York (NJ)" heads no State. "Washington, D.C." and "Washington DC" name the District in headings as well as claimant lines. Low 4: the tree builder follows HTML for more tags: - a button start ends an open button; - an li, dd or dt start ends the open item unless a special element other than address, div or p is open inside it; - a table start directly in a table ends that table; - caption, colgroup and section starts clear back to the innermost table, and so does a col outside a colgroup; - a row or cell start ends an open caption or colgroup of its own table; - a row or cell directly in a table gets an implied tbody, and a cell directly in a section an implied row; - a heading end tag ends the open heading in scope, through a button; - </br> is a br, and </body> and </html> close nothing; - </table> ends the innermost table, and a table part's end tag stops at its table; - </p> closes a paragraph in button scope, or else inserts an empty one; - </li> closes in list item scope, and other special end tags in scope; - any other end tag stops at a special element. Notes: _cell_token's br sentence now says what holds, the col and not-modelled wording follow the code, and the PR's Tests section is recounted. Evidence, outside the repository: - Every registered reader's output, cell values included, is byte-identical to 697ea34's. - Against html5lib 1.1, on 5,000 generated pages each: this round's generator gives 0 heading and 0 table mismatches (697ea34: 1,561 and 3,969), and round 6's gives 0 and 0. Round 4's gives 50, all on pages with search or dialog elements, which html5lib 1.1 predates (697ea34: 59). - On the 40 pinned captures, with comments and noscript removed from both trees, tables, block texts and parent-child nesting equal html5lib's on all 40 (697ea34: 40, 40 and 32). - In the pinned captures, 42 cells have a digit on each side of a br and 39 within one paragraph; none joins two amounts by a br within one paragraph. - 46 mutants, one per new or changed rule and including round 6's equivalents, are all caught. 51 of the 157 tests in test_aggregate_poms_invented.py fail against 697ea34's two modules. Tests: D4's six modules give 432 passed (176 unit, 256 artifact). The birth-evidence seal tests give 12 passed; the reachability test needs modules this sparse clone leaves out, so CI runs it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The round-7 delta review (GPT-6.1 Sol, at 4aae282) asked for revision. Its three medium findings were State-guard bypasses: - M1: a block whose whole text is a State date line skipped its lines, so a later line could date or name another State ("Nevada Effective ...<br>Utah Effective ..." kept Nevada with two date options). - M2: a claimant line that named another State passed whenever it also named the section's own ("Couple for Utah (see Nevada)"). - M3: an effective-date sentence other than "State Effective ...", such as "Utah payment levels effective ...", carried another State's date into the section, and so did a D.C. alias or a postal suffix before "Effective". The fix: - Every line of a block passes the State guards. When the whole text is a State date line, its lines still do not head a section. - A claimant line, and every effective-date sentence, may name no State but the section's own. Before the first State heading neither is guarded, as before: no section is open, and the heading that opens one sets its own date and claimant. - A State date line is recognized by the names, D.C. aliases and postal suffixes a section heading takes, through the helpers the heading reader now shares (_state_name, _state_reference). Low findings: - L1: within a line, inline elements now join without a separator, as HTML renders them, so U<span>tah</span> reads "Utah" and Cou<span>ple</span> "Couple". normalized_text, which the tree tests use as a fingerprint and the table reader for labels, still separates every text node; the docstrings say so. - L2: any line shaped as a State heading whose postal code is not that State's refuses, wherever it sits in the block ("Invented note<br>New York (NJ)"). A later line that only mentions a State in prose is still admitted. - L3: the three tree mutants that survived now fail tests: list items around an address, dd and dt ending each other, and </html> after an html start tag, with the expected blocks html5lib 1.1 builds. - Note: the tree's docstring lists a form start tag inside an open form among what it does not model, with the review's example. No pinned capture opens a form inside another. The pinned 2011 California section has "charts<span class="poms-bold">:</span>", which the old line reader read as "charts :". Its two charts' parameter_effective_date_text in poms_registry.json now read "charts:". Nothing else changes: every registered reader's output, cell values included, hashes as at 4aae282, and only the 2011 page's selector digest moves. Checks: - 18 of the 187 invented POMS tests fail against 4aae282's reader, the same 18 in each of three Hypothesis seeds. The new tree cases and the later-claimant-line case pass there: 4aae282 already built those trees and refused that line, and the cases catch mutants of it. - Mutants: the 14 for this round (G1-G11 for the guards, O1-O3 the review's tree mutants) and the 41 from earlier rounds, rerun on the final code, all fail tests. One earlier mutant, P11 (the claimant guard stops at its first claimant line), passed at first: a whole text naming another State now refuses on its own. A block too long to read as one claimant line, with its own State's claimant line first, now catches it. - On the 40 pinned HTML captures, the tree's tables, block texts and nesting equal html5lib 1.1's on all 40; on 5,000 generated pages per mode there is no heading or table mismatch. - D4's six test modules: 462 passed (unit 206, artifact 256). - black -l 79 and ruff are clean. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The round-8 delta review (Opus, at 93824c2) asked for revision, and CI failed on Python 3.14.7. Python 3.14: the standard library's HTMLParser reads xmp, iframe, noembed and noframes as raw text in 3.14.7 and not in 3.13.9, turns everything after a plaintext start tag into text in 3.14.7 only, and closes the empty comments "<!-->" and "<!--->" at different places. The tree now pins HTML's six raw text elements on its subclass (noscript stays markup, as in HTML with scripting off), refuses a page with a plaintext start tag or an abruptly closed empty comment, and closes the parser at the end of the page, so raw text left open keeps its text. The paragraph-closer test expects plaintext's page to refuse and still states the closer rule. The tests pass on both versions, the differential finds no mismatch on either, and 20 comment and raw-text edge pages build the same tree on both. Medium 1: a State name glued to an inline element's text became another word, so no State net saw it. Each block's whole text and lines are now read three ways: joined (inline text joins, as HTML renders it), bare (the joined reading without footnote markers, a sup or a link to a footnote, as table cells are read) and spaced (a space at every element edge). The joined and bare readings recognize a State heading and a claimant, so Utah<sup>1</sup> heads Utah; the joined reading alone gives the date sentence, because reading it bare moved one pinned chart's date. Every net runs over all three readings: another State's heading line, the misnamed heading, the State date line and effective-date guard, the claimant guard and the heading net. A claimant line that only the spaced reading shows refuses. Medium 2: when a block's whole text is a State date line, a line shaped as another State's heading now refuses, whatever the whole text's length. The test docstring that said a long block's lines refuse now says what the reader does, with three long_heading_line cases. Low 3: the review's R1, R3 and R5 witnesses are tests, and each mutant fails them. Low 4: a block whose claimant lines name two claimant types refuses, and the claimant resets after each chart, so every chart needs its own claimant line. No pinned chart loses its claimant. Low 5: every line of an h1-h6 heading is netted, in every reading, for a State other than the section's. This reverses round 8's admission of "Invented note<br>Invented note on Nevada" in California's section; a later heading line naming the section's own State, and paragraph prose naming any State, are still admitted. Low 6: the District alias takes "Washington,DC" and "Washington,D.C.". Note 7: every reading drops Unicode format characters (category Cf, 170 code points, the same set in Unicode 15.1 and 16.0), the soft hyphen included. "Hawaiʻi" and its apostrophe spellings name Hawaii and may head its section. A postal code in parentheses in either case, an upper-case postal code standing as a word and fourteen States' abbreviations, "N.J." among them, name their States to the nets but never head a section. Abbreviations without their period, lower-case codes outside parentheses and hidden text are listed under "Not modelled". Note 8: a later html start tag builds nothing and adds only the attributes the html element lacks, as html5lib does. Extending the differential to raw text, noscript and later html start tags found two more tree gaps, now closed: a noscript end tag with a special element open inside it is ignored, and a table part's start tag outside any table builds nothing. Checks: - Every registered reader's output, cell values included, hashes as in round 8 on Python 3.13.9 and 3.14.7 (registry-v2-r8.json, byte for byte). All 777 pinned charts' selectors equal 93824c2's. - On the 40 pinned HTML captures, the tree's tables, blocks and nesting equal html5lib 1.1's on all 40, on both versions. - html5lib differential: no mismatch on 5,000 pages per mode with round 7's generator, or with this round's, which adds raw text, noscript, later html start tags and plaintext (626 pages refused, 9,374 compared), on both versions. - D4's six test modules: 575 passed on 3.13.9 and on 3.14.7 (unit 319, artifact 256; round 8 had 462, unit 206). - Mutants: 59 of this round's 60 fail tests. H1, which nets a heading's whole text and first line only, survives: the whole text holds every line's text, so only an alias split across lines tells it apart. mutants_r8.py: 7 of the 7 of its 14 that still apply. mutants_r7.py: 31 of the 31 of its 46 that still apply. This round ports the rest, or names the round-8 or round-9 mutant that makes the same change. - On Python 3.13.9, 92 of the 300 invented POMS tests fail against 93824c2's reader, the same 92 in each of three Hypothesis seeds. - black -l 79, ruff and git diff --check are clean. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The svg and math end-tag check dropped its "more than one ended" branch: an end tag that names the innermost svg or math closes only that one, so the branch never decided a page. The tests add a short line holding each dating word alone, a split date that only the bare reading shows, and an h5 whose earlier year holds no dating word, so that a mutant of each of those rules fails a test. The earlier-year test drops two cases that held a second dating word. Pinned output holds: 28 files the same, 0 differ, 0 refused, 777 charts, and the registry fingerprint matches round 8's byte for byte. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Under the round-13 footnote-word rule the review's Q7 witness, a leading "See also" sup, refuses with or without the mutant. A digit marker before a leading "As" sup separates them: every dropped word must be a footnote word, so the line is no footnote body and its date leaves the section's unread, while Q7, which needs any one word, makes it a footnote body. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Three sentences that the round-13 fix added or extended now name who acts: aggregate_tables lists the tree's gaps, a long date sentence drops its second date, and the tree pops the svg or math whose place the stack has lost. Both modules' ast, without docstrings, equals 179496e's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Round 13 found that reading "Effective Dates:" in any case (1b319dc) turned pages the parent read or refused into silent wrong reads. The three lines go back to 32cc85d's text. A long sentence that holds the words in lower case dates its chart and meets the State guards again, and a range in another case sets no section revision date. Round 12's Note 4 returns as a stated limit. The round's other findings reach invented pages only and change no record on a pinned input, so they stay as stated limits. The docstrings of both readers stop claiming those forms, and aggregate_tables.py changes in docstrings only. Tests replace the four cases that pinned the any-case reading with the review's E4, ES1, V6 and V7 and the lower-case revision line. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Master at 1808285 holds unit 19,664, artifact 4,271, integration_psid 1,471, reproduction_legacy 520, oracle_policyengine 308, total 26,234. D4 adds 794 unit and 256 artifact tests, so the files now read unit 20,458, artifact 4,527, total 27,284. CI's tier policy test checks the count on the full collection. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis
marked this pull request as ready for review
November 2, 2026 09:02
This was referenced Nov 2, 2026
Keep both opt-in blocks in the birth-evidence exclusions and its reachability test: SSI0 from master and the aggregate-table readers from D4, with the shared package __init__ listed once. Take master's tier files; the next commit recounts them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Master at 89d064b holds unit 20,454, artifact 4,271, integration_psid 1,471, reproduction_legacy 520, oracle_policyengine 337, total 27,053. D4 adds 794 unit and 256 artifact tests, so the files now read unit 21,248, artifact 4,527, total 28,103. CI's tier policy test checks the count on the full collection. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Master adds #563's Track B changes. Only the two tier files conflicted; take master's, and the next commit recounts them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Master at edba88f holds unit 22,859, artifact 4,271, integration_psid 1,471, reproduction_legacy 520, oracle_policyengine 337, total 29,458. D4 adds 794 unit and 256 artifact tests, so the files now read unit 23,653, artifact 4,527, total 30,508. CI's tier policy test checks the count on the full collection. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
Author
|
The Microcosm Dynamics coordinator merged this PR at
|
MaxGhenis
added a commit
that referenced
this pull request
Nov 3, 2026
Master's #573 (SSI0) and #574 (D4) each added a block to the birth-evidence exclusions, their expected tuple and the reachability test, where D2 adds its own. Keep master's blocks and put D2's after them in all three places. Both sides add the same empty tests/income_resources/__init__.py. The merge takes master's tier files, and the next commit recounts them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Nov 3, 2026
MaxGhenis
added a commit
that referenced
this pull request
Nov 3, 2026
Master's #573 (SSI0) and #574 (D4) each added a block to the birth-evidence exclusions, their expected tuple and the reachability test, where CE1 adds its own. Keep master's blocks and put CE1's after them in all three places. D4's block already lists src/populace_dynamics/income_resources/__init__.py, which both sides add as the same blob, so CE1's exclusion blocks keep only coverage_rules.py. The merge takes master's tier files, and the next commit recounts them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Nov 3, 2026
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is part 1 of milestone D4 of the income-and-resources design (#559: the D4 row of the milestone table). It adds pinned captures and parsers for aggregate tables. Part 2 adds the SSA and BLS sources that refuse plain requests, through Internet Archive captures. Thirteen independent review rounds asked for fixes.
8cb397f5, on17bdcf00, fixes round 13's one medium finding, a regression against the parent. The round's low findings change no record on a pinned input, so they stand as the limits listed under Open items, and the claims they falsified now say less. The round-14 delta review approved8cb397f5. Since then we merged master into the branch four times and recounted the tiers three times. Only the merge of89d064b5conflicted outside the tier files; it combines D4's birth-evidence exclusions and reachability checks with #573's SSI0 additions.What it adds
data/external/snapshots/income_resources_d4_v1/: 54 pinned sources. Each per-source manifest records time, size, SHA-256 and URL. A folder-level.gitattributes(* -text) keeps the 37 captures with CRLF line endings byte-exact.src/populace_dynamics/income_resources/aggregate_*.py):Invariants
None, never zero.TableStructureError, on every form its module docstring says it reads. The docstring's Not modelled, Limits and leave-standing lists name the forms it does not read that the review probes found, among them an abbreviation without its period, text outside any block or in a chart's owncaption, a long line of prose, an effective-date sentence of 80 characters or more whose second date takes another form, a claimant named in other words than its three claimant words, a date with a two-digit year in a form no sentence reads ("January 1, 31"), a date split across two blocks, a date split across a<br>in a block of 80 characters or more, a date in a form no list names ("July 1", "7/1", "July of 2030"), and a short line outside a heading that holds only years before the dates in force and no dating word. The tests pin the round-12 review's table rows and witnesses and many of the listed limits; neither the tests nor the lists show that no other form exists.<br>and where a nested block starts or ends, and drops Unicode format (Cf) characters and the text ofscriptandstyle, which HTML does not render, a footnote marker's text included. The reader reads each block's whole text and each line five ways:sup, or a link to an anchor on the page (href="#..."), whose text is one to three word characters or footnote signs. A longersup, or a link elsewhere, is text, so no word of prose vanishes;supor link whose href holds "fn".Utah<sup>1</sup>heads Utah. Every State net reads all three readings, so no inline edge hides a State.January 1<sup>2</sup>, 2031,January 1<a href="0501415010#note5">5</a>, 2031) changes no date silently; the page refuses. A line that is a dated sentence only in its bare, unmarked or unlinked reading refuses too, unless what those readings drop all leads it, as a footnote body's markers do (<sup>1</sup>Effective ...), which dates nothing. A block's whole text is its sentence when it is one; otherwise each line that is one dates the block, and lines that date differently remain named alternatives. A range is "effective from", a date, "to" or "through", and a date; it dates a chart from and to, in its own order, when its two dates are the only ones the sentence states, and other dates remain named alternatives. A "to" that no date follows ends no range, and a range that does not end after it starts refuses, one from a date to the same date included.<br>, though a revision range on a line still counts among the lines. Outside a heading, a footnote body dates no chart: a line, or a block's whole text, that opens with markers by either rule, every word they drop one to three digits or footnote signs or a single letter.<br>as within a line. A heading, or a line under 80 characters, that holds a claimant word or plural but is no claimant line ("Couple payment levels:", "Eligible couple", "Couples") names a claimant the reader does not read: the next chart refuses unless a later claimant line names one, and one that holds both types refuses as naming two. Prose of 80 characters or more outside a heading names none, though the State guard reads it. Every chart needs its own claimant line.<svg><foreignObject><g></svg>HTML ignores the end tag and keeps the svg open, while the tree ends it. A CDATA section holds no text in the tree, though HTML reads one inside svg or math as text. The tree reads a page the same way under Python 3.13 and 3.14: it pins raw text to HTML's six raw-text elements (script, style, xmp, iframe, noembed and noframes; noscript stays markup, as with scripting off), and a plaintext start tag or an abruptly closed empty comment refuses, and so does a script whose text holds<!--and then a script start tag, which HTML's script data escape states may run on past its next</script>, a self-closed raw text or RCDATA start tag (<script/>), whose slash HTML ignores outside svg and math, and a raw text or RCDATA element inside svg or math whose text holds<, which HTML reads as markup there. A later html start tag only adds the attributes the html element lacks, the tree ignores a noscript end tag with a special element open inside it, and a table part's start tag outside a table builds nothing, as in html5lib.poms_open_questions.jsonrecords the chart. We infer no definition.Open items
_HTMLTree,_cell_tokenand the POMS module:<h3>w0<b><i>w1</b>w2<h4>w3, HTML's h3 reads "w0w1w2w3" and the tree's "w0w1w2"), foster parenting of content misplaced in a table, leading content HTML moves into an implied head, a self-closing slash on another non-void element, or foreign content. So it refuses a page on which an element other than svg and MathML ones starts inside svg or math, breakout tags and HTML inside svg's foreignObject or MathML's mi included, on which svg or math ends other than at its own end tag, or whose script, style or title there holds<, and it refuses any self-closed raw text tag. The stated limits below name what that rule misses. The 40 pinned HTML captures hold 159 svg icons of path, title and desc elements and no math, so no pinned page refuses.<br>or nested block in one cell joins its lines with a space, so two amounts stacked that way would read as one space-grouped number. No pinned cell joins two amounts by a<br>within one paragraph. POMS charts stack amounts in paragraphs, which the reader splits into subrows.hiddenattribute, for one), as text. They never read a caption in a table cell, a chart's owncaptionelement or text outside any block. They over-refuse where an acronym spells a postal code, as "MA" or "IA" in Pennsylvania's section, on a short line that names another State anywhere, on a heading or short line that names two claimant types or a date no sentence states, on a heading that holds a year outside the dates in force, on a short line that holds a later year, or an earlier one with a dating word ("Separate programs for 2029", whose "Separate" holds "rate"), on a number shaped as a dotted date ("Section 2.1.10"), on a footnote marker of two letters or of a digit and a letter, on a heading that opens with a footnote marker, on a link of a marker's size inside a date, and on a range from a date to the same date. No pinned page refuses.<svg><foreignObject><g></svg>, and the same shape in desc, title, mi, mo, mn, ms, mtext or annotation-xml, ends the svg in the tree while HTML keeps it open, so later blocks and cells read here and stay foreign in HTML.<svg><text><![CDATA[Couple]]></text></svg>reads "Couple", and in HTML content reads the text after its first>.Effective Dates: 03/01/2031 - Present<br>Payment levels as of March 1,<br>2031keeps the section's date.<br>in a block of 80 characters or more, outside a heading, names no date unless the block is an effective-date sentence, so a split date in a year in force keeps the date before it.supof one letter or digit makes a footnote body even where it opens a word or a year, as<a href="#top">A</a> new payment level applies from March 1, 2031or<a href="#y">2</a>031 payment levels, so its date or year dates no chart.Effective dates: 03/01/2031 - Presentreads as a sentence and dates the chart, since the label reads in title case only.parse_html's raw-text check walks every svg's whole subtree, so nested svg parses in quadratic time: 0.27 s at 4,000 levels, and the review estimates 40 s at 50,000.aggregate_tables.pysays "HTML element other than svg or MathML" for svgaandfilter, which refuse because they sit outside_FOREIGN_VOCABULARY.SOURCE_BLOCKED), so we infer no coverage share.Review and fixes
The first review found that the POMS reader kept only the last definition table, so California 2004 lost codes A–F. The fix reads every definition table in the State's section. It also surfaced three more reader problems, now fixed:
new_jersey.The fix adds these checks:
Each of the four bugs, reintroduced, fails at least two tests. The birth-evidence seal's exclusions list the five new source files, with a reachability check.
The second review found that a definition table placed between two charts of one State section would have reached only the later chart. The source does not say which earlier chart such a table serves, so the reader now refuses it. No pinned capture has that order, and all 777 charts describe as before. The fix adds:
The third review found that the reader did not recognize a State heading in a paragraph left unclosed, such as
<p>Nevadafollowed by the chart's heading and table. The shared HTML tree never closes a paragraph implicitly, so the paragraph's text ran on into the chart, and the previous State's selector and definitions carried into the next State's chart. The reader now reads each paragraph or heading from its own text, up to the first nested element whose start tag closes a paragraph in HTML (we checked the set against html5lib 1.1). No pinned capture nests a block in a paragraph or heading (none of 68,336), and all 777 registered selectors, definitions and parsed cells keep their hashes. The fix adds:</p>;With the old reader, the regression and the omitted-ends property fail.
The fourth review found that this fix cut a heading's text at blocks that HTML keeps inside a heading, so
<h3><div>Nevada</div></h3>again kept California's definitions. It also found that an omitted</p>in a stacked payment cell merged its amounts, that a page of thousands of unclosed paragraphs exhausted the stack, and that the closer set lacked dialog and search. The fix (401b3bc7):Against html5lib 1.1, the new tree agreed on 20,000 random pages of paragraphs, headings and block containers; the old tree disagreed on 6,639.
The fifth review found one medium and three low problems:
<h3>Nevada<p>Invented note</p>, no longer matched the State pattern, so the previous State's selector carried on. Claimant headings failed the same way. The tree also ended a heading only at its own end tag, where HTML ends it at any h1–h6 end tag.The fix (
8613b313):colout: all 4,045coltags in the 28 pinned captures that have one sit in a colgroup;Nothing changes on the pinned captures: the tree comparison builds 44 identical trees, and all 874 registered entries hash identically. (Round 4's commit called these "43 HTML captures". The registry holds 41 HTML captures, and the comparison's byte filter also admits two xlsx workbooks and a zip archive.) Against html5lib 1.1, on 5,000 invented pages per mode, the new tree has no mismatch in headings (
401b3bc7: 506) or in tables with optional cell, row, section and caption ends (401b3bc7: 1,230). Each of 27 mutants of the tree and the POMS reader fails at least one test, as does each of five mutants of the cell reader.The fix adds, on invented HTML only:
The sixth review (delta, at
697ea340) found one medium and three low problems:</h3>sat._payment_rowsdeep-copied the chart, and ElementTree's copy recurses, so a cell nesting 50,000 spans crashed the interpreter. The recursion invariant above was therefore false.<br>, a zero-width character or another State's postal code was misread, and so was a "Washington, D.C." heading.The fix (
4aae2820):<br>and drop zero-width characters. A postal code after a State's name must be its own, and "Washington, D.C." and "Washington DC" name the District in headings;Nothing changes on the pinned captures: every registered reader's output, cell values included, hashes as at
697ea340. On the 40 pinned HTML captures, the tree's tables, block texts and nesting now equal html5lib 1.1's on all 40 (697ea340: 40, 40 and 32). That comparison removes comments and noscript from both trees. On 5,000 generated pages per mode, the tree has no heading or table mismatch with html5lib (697ea340: 1,561 and 3,969). Each of 46 mutants, one per new or changed rule, fails at least one test. 51 of the 157 tests intest_aggregate_poms_invented.pyfail against697ea340's modules.The fix adds, on invented HTML only:
</h3>, and a property that selectors do not depend on heading ends or block kinds;<br>-separated line that dates another State;<br>,</br>,</p>, zero-width characters, postal codes and Washington, D.C.;The seventh review (delta, GPT-6.1 Sol, at
4aae2820) found three medium and three low problems and one note:</html>after an html start tag.The fix (
89b40e46):The pinned 2011 California section's date sentence ends
charts<span class="poms-bold">:</span>, which the old line reader read as "charts :". Its two charts'parameter_effective_date_textinpoms_registry.jsonnow read "charts:". Every registered reader's output, cell values included, hashes as at4aae2820; only that page's selector digest changes. The html5lib comparisons hold as before: 40 of 40 pinned captures, and no mismatch on 5,000 generated pages per mode. Against4aae2820's reader, 18 of the 187 invented POMS tests fail, the same 18 in each of three Hypothesis seeds; the new tree cases and the later-claimant-line case pass there, since4aae2820already built those trees and refused that line. This round's 14 mutants (G1–G11 for the guards, O1–O3 the review's tree mutants) and round 7's 41, rerun on the final code, all fail tests. P11 (the claimant guard stops at its first claimant line) passed at first, because a whole text naming another State now refuses on its own; a block too long to read as one claimant line, with its own State's claimant line first, now catches it.The fix adds, on invented HTML only: the review's four repro fragments and their variants, refused or admitted, including long blocks whose whole text is no effective-date or claimant line, one with the section's own claimant line before another State's; controls that read the section's own State, date and claimant; an effective-date line before any State heading; and five tree pages whose expected blocks are html5lib 1.1's.
The eighth review (delta, Opus 5.5, at
93824c2c) found two medium and four low problems and three notes, and CI failed on Python 3.14.7:sup, from every State net;4aae2820had refused those pages.The fix (
dfb30420):Every registered reader's output, cell values included, hashes as at
93824c2con both Python versions, and all 777 pinned charts' selectors equal93824c2c's. On 3.13, 92 of the 300 invented POMS tests fail against93824c2c's reader, the same 92 in each of three Hypothesis seeds.mutants_r9.py's 60 mutants fail tests but one, H1, which nets only a heading's whole text and first line; the whole text holds every line's text, so it differs only for an alias split across lines. Of earlier rounds' mutants, the tests catch all 7 ofmutants_r8.py's and all 31 ofmutants_r7.py's that still apply; we ported the rest intomutants_r9.py, or they match a caught mutant. (Round 8's "round 7's 41" meant the 41 ofmutants_r7.py's 46 that applied then.)The round-9 build listed a prose line holding a claimant word, such as "A recipient or couple qualifies if ...", as able to set the next chart's claimant, and counted 70 such line forms on the pinned captures. That overstated the exposure. Those 596 lines are the block lines that hold a claimant word and then "or" under the old pattern. The 592 in table cells, which the claimant reader never reads, each hold two claimant words, one of each type; the other 4 are one h4 heading form with one claimant word and the source's "or" for "for", a claimant line under the new rules too. Of the 777 pinned charts, 594 take their claimant from h1–h6 lines, 99 from short whole-text paragraphs and 84 from State date captions; none takes it from prose. Round 10 makes prose name no claimant (below), and all 777 charts read as before.
The ninth review (delta, Opus 5.5, at
7adafa66) found three medium and two low problems and seven notes:January 1<sup>2</sup>, 2031read January 12) or hid the sentence, silently.sup, and any link whose href held "fn", so link orsuptext vanished and prose read as another State's heading. Round 8 introduced this.<p>Utah (continued)</p><p>Couple</p>charted Nevada.The fix (
8a772f5a):supor#link holding one to three word characters or footnote signs;_cell_tokenkeeps its wider rule;Every registered reader's output, cell values included, hashes as at round 8 on both Python versions, and all 777 pinned charts' selectors equal
7adafa66's; no pinned page refuses. On the 40 pinned HTML captures the tree's tables, blocks and nesting equal html5lib 1.1's. On 3.13, 94 of the 434 invented POMS tests fail against7adafa66's reader, the same 94 in each of three Hypothesis seeds. All 41 ofmutants_r10.py's mutants fail tests: at least one per new rule or guard, and ports of the round-9 and round-7 mutants whose snippets this round removed. A mutant of the marker pattern's three-character bound alone survived, because the marker walk capped text at three characters too; both now read one constant, and the tests catch its mutant. The review's O4 is not equivalent: a footnote body opening a State date line, followed by a line shaped as that State's heading, refuses on the head and reads under O4, and a test now pins it. Of the older sets, the tests catch the mutants whose code still exists: 5 of 5 fromown_mutants_r9.py, 53 of 54 frommutants_r9.py(H1 survives, as in round 9), 7 of 7 frommutants_r8.pyand 30 of 30 frommutants_r7.py.The tenth review (delta, Opus 5.5, at
8a772f5a) found two medium and four low problems and six notes:<h4>Individual<br>or couple</h4>read as individual and<h4>Couple<br>Eligible individual</h4>as couple.January 1<sup>5<script>x</script></sup>, 2031read January 15).<!--<script>ended at its first</script>in the tree, though HTML runs it on.The fix (
85958483):<!--and then a script start tag;caption, text outside any block, the script escape states and the new date refusals.Every registered reader's output, cell values included, hashes as at round 8 on both Python versions, and all 777 pinned charts' selectors equal
8a772f5a's; no pinned page refuses. On the 40 pinned HTML captures the tree's tables, blocks and nesting equal html5lib 1.1's. On 3.13, 94 of the 548 invented POMS tests fail against8a772f5a's reader, the same 94 in each of three Hypothesis seeds, over 19 test functions. Ofmutants_r11.py's 49 mutants (at least one per new rule or guard, and ports of the earlier sets' mutants whose code this round rewrote), 40 failed tests at85958483, and nine passed all six modules, one of them (R21) equivalent.858fb546adds 15 test cases on invented pages, one to three per survivor; at that commit 48 of the 49 fail tests, each of the eight first failing its own witness. R21 walks script children whose text no reading uses, so no test can tell it apart. Every mutant ofmy_mutants_r10.py,mutants_r10.pyandown_mutants_r9.pywhose code still exists fails tests (8, 30 and 3), M4, M16, M20 and M21 among them; the review's H1 differs only where the reader over-refuses, so no test can kill it without pinning that over-refusal. Rerun at858fb546, the three oldest sets give: ofmutants_r9.py's 60, 50 fail tests, 9 change code that no longer exists and H1 passes all six modules; ofmutants_r8.py's 14, 7 fail tests and 7 change code that no longer exists; ofmutants_r7.py's 46, 30 fail tests and 16 change code that no longer exists; the W2a and W2b ports fail tests too.The eleventh review (delta, Opus 5.5, at
858fb546) approved it with six low findings and seven notes:supor link whose href holds "fn" for a footnote marker, so short lines such asPayment levels as of March 1, 2031<br><sup>1</sup>Revised March 1, 2031kept the section's earlier date silently. That made the Invariants' "never reads a wrong one silently there" false.mutants_r7.py's results (all corrected above); the tree's docstring said HTML's escape states run a script on where they may not.<script/>hides what follows in HTML but not in the tree; a short line that repeats the claimant in force refuses; a claimant named in other words keeps the earlier type; a range to the same date drops its end and a reversed range reads literally; padding can push a short line past 80 characters; process.The fix (
8efc4b65, with tests in31ae38efand docstring wording in5bc9cb6f, and witness cases in32cc85d0):<in its text, and a page with any self-closed raw text or RCDATA start tag; a census finds neither in the 40 pinned HTML captures, and the tree's docstring lists foreign content under Not modelled;All 777 pinned charts' selectors equal
858fb546's on 3.13 and 3.14 (28 files the same, 0 differ, 0 refused, 777 charts, on each), and every registered reader's output, cell values included, hashes as at round 8 (874 specs, byte-identical on 3.13 and 3.14). 61 of the 667 invented POMS tests at5bc9cb6ffail against858fb546's readers, over 15 test functions, none of them one858fb546defines. Ofmutants_r12.py's 45 mutants (29 for the new rules, 15 from the review's set ported, with X4 split into X4 and X4b for the whole text's bound), 44 fail tests at5bc9cb6f; S24, which let any four digits count as a year, passed all six modules until32cc85d0added witness cases. Rerun on5bc9cb6f's readers, which32cc85d0leaves unchanged, the older sets give: ofmutants_r11.py, 34 fail tests, 14 change code that no longer exists and R21, the equivalent one, passes; ofmutants_r10.py, 28 fail tests and 13 change code that no longer exists; ofmy_mutants_r10.py, 7 and 3; ofmutants_r9.py, 49 and 9, H1 passes for the reason round 11 checked, and V1 now stops the module loading, since_TEXT_CONTENTreads the set it deletes; ofmutants_r8.py, 7 and 7; ofmutants_r7.py, 30 and 16; and W2a and W2b fail tests. Of round 9's own set, on the same readers, 3 fail tests and 3 change code that no longer exists.The twelfth review (delta, at
32cc85d0) approved it with five low findings and six notes:<svg><section>,<svg><td>and others) as HTML, where HTML keeps them foreign, and ended an h5 at an end tag inside MathML'smi, which HTML ignores there, so a date, State or claimant dropped out of its block, and a chart row's amounts moved one column on.supor footnote link held for a marker, so<sup>As of</sup> March 1, 2031and<a href='#new'>New</a> payment levels as of March 1, 2031dated no chart.<script src='x.js'/>, is the common real form.<br>, a heading that opens with a marker); a lower-case "Effective dates:" line dated a chart; the new over-refusals; process.The fix (
1b319dca, with tests inbe276733and179496ecand docstring wording in17bdcf00):All 777 pinned charts' selectors equal
32cc85d0's on 3.13 and 3.14 (28 files the same, 0 differ, 0 refused, on each), and every registered reader's output, cell values included, hashes as at round 8 (874 specs, byte-identical on 3.13 and 3.14). All 40 pinned HTML captures parse; their 159 svg hold only path, title and desc elements, and none holds math. 72 of the 774 invented POMS tests at17bdcf00fail against32cc85d0's readers, over 12 test functions, none of them one32cc85d0defines. Ofmutants_r13.py's 50 mutants (38 for the new rules and 12 from the review's set, ported), 49 fail tests at17bdcf00, and Q2, the equivalent round 12 found, passes. In process, 49 of the 50 change the outcome of a page their tests build, all but Q2. Rerun at17bdcf00, the older sets give: ofmutants_r12.py, 31 fail tests and 14 change code that no longer exists; ofmutants_r11.py, 33 and 15, and R21, the equivalent one, passes; ofmutants_r10.py, 28 and 13; ofmy_mutants_r10.py, 7 and 3; ofmutants_r9.py, 49 and 9, H1 passes for the reason round 11 checked, and V1 stops the module loading; ofmutants_r8.py, 7 and 7; ofmutants_r7.py, 30 and 16; and W2a and W2b fail tests. Of round 9's own set, 3 fail tests and 3 change code that no longer exists. Of the round-12 review's 12, 8 fail tests, 3 change code that no longer exists and Q2 passes, as above.be276733's message says two removed test cases each held a second dating word; "Effective in 2029" held one.The thirteenth review (delta, at
17bdcf00) returned REVISE with one medium finding, six low findings and seven notes:1b319dcaanswered round 12's Note 4 by reading "Effective Dates:" in any case. A long sentence that held "effective dates:" then stopped dating its chart and stopped meeting the State guards, so a page that named Utah in Nevada's section read as Nevada, where the parent refused. A range in another case, in script text or a table cell, set the section's revision dates.</svg>, so later blocks and cells stay foreign while the tree reads them. In a browser the same holds for desc, title and the MathML token elements.Under the project's review exit rule, a regression against the parent gets a fix, and an invented-input finding with zero effect on pinned inputs stands as a stated limit. So only Medium 1 changes code. The fix (
8cb397f5):_effective_line's exclusion,_REVISIONand the "Present" check read title case again, the three lines as32cc85d0wrote them. Round 12's Note 4 returns as a stated limit. Each of the 29 pinned POMS pages holds a single revision range, in title case;All 777 pinned charts' selectors equal
32cc85d0's on 3.13 and 3.14 (28 files the same, 0 differ, 0 refused, on each), and every registered reader's output, cell values included, hashes as at round 8 (874 specs, byte-identical on 3.13 and 3.14). All 40 pinned HTML captures equal html5lib 1.1's tree. The review's 16 regression probes read alike on8cb397f5and32cc85d0.aggregate_tables.py's syntax tree matches17bdcf00's, docstrings aside, so the html5lib differentials below stand. 68 of the 775 invented POMS tests fail against32cc85d0's readers, over 10 test functions, none of them one32cc85d0defines, and 5 fail against17bdcf00's, over the four new functions. Ofmutants_r13.py's 50 mutants, 46 fail tests, R1, R2 and R3 change code that no longer exists, and Q2, the equivalent round 12 found, passes. The review's twelve mutants pass all six modules, as it predicted. Onlyaggregate_poms.py's three lines changed, so we did not rerun the older sets.A census of the pinned captures backs the limits. None of the 40 HTML captures holds a CDATA section. Each of the 29 POMS pages holds one revision range, in title case, and no "effective dates:" in another case. None of the listed date forms appears on any of the 4,263 lines under 80 characters that the reader's own
_linesgives outside a table cell. The net also reads heading lines of any length: outside tables, in four readings, 14 lines match the date net's shape, each a page-title h1 of 97 to 100 characters before the first State heading, where no net runs.The fourteenth review (delta, at
8cb397f5, an Opus 5.5 reviewer after the Sol lanes closed) approved, with no high or medium finding, one low finding and five notes:8cb397f5restores the three lines round 13's medium named, each equal to32cc85d0's text. On any page, the head gives32cc85d0's read or refuses, and the review's own runs read 28 files the same, 0 differ, 0 refused and 777 charts on 3.13 and 3.14._REVISIONcaptures only "Present" or a date. Note 5 records the reviewer's own two runs above the load gate.8cb397f5.The earlier slot turn merged master
18082859(ce188f20) and recounted the tiers (5285c76d). Master's tier files replaced the branch's, and the recount added D4's 794 unit and 256 artifact tests.tests/tier_counts.jsonthen read unit 20,458, artifact 4,527, integration_psid 1,471, reproduction_legacy 520 and oracle_policyengine 308, 27,284 in all. CI's tier-policy test at8cb397f5collected those unit and artifact counts on the PR's merge with master, against a manifest of 19,664 and 4,271 (run 41344829714).A comparison of blob hashes from
8cb397f5to5285c76dfinds 46 moved paths: the two tier files and 44 other paths that master changed since the merge base0f69bbb2. All 44 match master's blobs. None of D4's 79 paths moved, and no path changed on both sides.CI at
5285c76dpassed every check but one test in shard 3,tests/test_track_b_b0_2_counts.py::test_a_failure_from_outside_this_script_prints_no_value, which ran first in that shard. CI at8cb397f5failed the same way. #596 ("Flush the real streams before each B0.2 test captures output") merged ahead of this branch asdd9c7be2, and35859218carries it. CI at35859218passed every check, shard 3 included.ecc09804merges master89d064b5, which brings #573's SSI interface. #573 and D4 each added a block to the same exclusion tuple inscripts/first_estimates_birth_evidence.py. They also each added a block to the same reachability test intests/estimates/test_birth_evidence_artifact.py, so both files conflicted. The merge keeps both blocks and lists the sharedincome_resources/__init__.pyonce, in D4's block. Against master, it adds the 7 and 20 lines D4 added to these files, and it drops only master's copy of that__init__line.d5dd388frecounts on89d064b5: unit 21,248, artifact 4,527, integration_psid 1,471, reproduction_legacy 520 and oracle_policyengine 337, 28,103 in all.35859218merges masterdd9c7be2, which changes one test module for #596 and leaves the tier manifests unchanged, so the recount stands.From
5285c76dtod5dd388f, 159 paths changed: the two tier files, the two conflicted files and 155 other paths that master changed since18082859. All 155 match master's blobs. None of D4's other 77 paths moved. Fromd5dd388fto35859218, one path moved: the test module #596 changed, now equal to master's blob.38895135merges masteredba88f7, which brings #563's Track B changes and adds 2,405 unit tests to master's count. Only the two tier files conflicted, and the merge takes master's.ca93c707recounts onedba88f7: unit 23,653, artifact 4,527, integration_psid 1,471, reproduction_legacy 520 and oracle_policyengine 337, 30,508 in all. From35859218toca93c707, 63 paths changed: the two tier files and 61 other paths that master changed sincedd9c7be2. All 61 match master's blobs. None of D4's paths moved.Tests
D4's six modules (
test_aggregate_coverage,_numbers,_pensions,_poms,_poms_inventedand_ssa): 1,050 passed on Python 3.13.9 and 1,050 on 3.14.7, up from 944. On the merge of89d064b5, all 1,050 passed again on Python 3.13. The birth-evidence seal tests' reachability test needs modules this sparse clone leaves out, so CI runs it, as it runstests/track_u2/test_u2_isolation.py.black -l 79andruffare clean.tests/tier_counts.jsonandtests/README-tiers.mdcarry the recount on masteredba88f7: unit 23,653, artifact 4,527, total 30,508.858fb546). Round 11's generator, round 10's with the script escape form added from its own random stream, refuses each page whose script html5lib runs on (32 headings and 72 table pages per 5,000) and finds no mismatch on pages without a formatting element left open or misnested (3,793 headings and 2,397 table pages compared); on pages with one, the differences are the adoption agency and reopening rules the tree does not model. The differential counts apart the pages that differ only by a newline right after a pre, listing or textarea start tag in a table cell or caption, which the tree drops, as it does everywhere, and html5lib 1.1 keeps there (18 headings and 90 table pages per 5,000). Round 12's generator, round 11's with svg and math content and self-closed raw text tags added from a third random stream, refuses 149 headings and 555 table pages for raw markup inside svg or math and 140 and 639 for a self-closed tag, and finds no mismatch on the 3,507 headings and 1,869 table pages with no formatting element left open or misnested and no svg or math breakout. Of the 132 headings and 380 table pages with breakout markup, 19 and 83 differ. Of the 19 heading pages, 9 are breakout, each a<math><h4>inside an h3, h4 or h5, which html5lib moves out, and 10 differ by the adoption agency rules alone; all 83 table pages differ by the adoption agency rules alone. At the reader's level, the 8 breakout pages with no formatting markup read the same, and page 919 differs only by an extra emptyp. Round 11's generator gives the round-11 review's logs byte for byte on the round-12 tree. Round 13's generator, round 12's with the review's foreign forms added from a fourth random stream (<svg><section>and the other block names HTML keeps foreign,<svg><td>in a cell, and<math><mi>before a heading end tag), draws round 12's pages unchanged where it draws no round-13 form. On the round-13 tree it refuses 616 headings and 2,598 table pages because an element other than svg or MathML starts inside svg or math, 107 and 139 of them round 12's breakout pages, which the tree no longer reads. It refuses 181 and 79 because svg or math ends other than at its own end tag, and finds no mismatch on the 3,104 headings and 1,201 table pages with no formatting element left open or misnested. Of the 637 headings and 507 table pages with one, 133 and 152 differ, by the adoption agency and reopening rules. The 4 headings and 16 table pages it draws with breakout markup that the tree still reads hold no svg or math in html5lib's tree, because raw text took the markup. One of those table pages differs by the adoption agency rules, on a misnested<b><code></b></code>ahead of the noframes that holds its svg, and another differs only by the newline html5lib 1.1 keeps, one of 12 headings and 20 table pages that differ that way. Round 11's generator gives the round-11 review's logs byte for byte on the round-13 tree.axiom: n/a: pinned public aggregate tables and their readers for microsim validation; no PolicyEngine country-model or RuleSpec encoding changed
🤖 Generated with Claude Code