diff --git a/AGENTS.md b/AGENTS.md index 3114a1acc..28e9b1b24 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -267,7 +267,7 @@ The 2.0 rewrite lands as underscore-private modules alongside the v1 code. These **Adding a curated sub-set of an existing vocabulary set** (a marker over a base set, as `AFTER_GIVEN_TITLES` is over `TITLES`) — it is a behavioral PROPERTY of a reading, not a second word list: a row in `tools/vocab/build.py`'s `BEHAVIORAL` table, a branch in `compile_pool` adding the words whose reading carries it, a constant in its `CONSTANTS` roster and a key of the same name in its `Compiled` TypedDict (an import-time assert holds the two to one list), and the `NAME = _compiled.NAME` assignment in the config module. The subset relation then holds by construction, since the property cannot exist without its reading, so it needs no check. A marker that records a clash with a NAME is the other shape: derived from the word's name reading beside the clashing one (`SUFFIX_ACRONYMS_AMBIGUOUS`, `SUFFIX_WORDS_AMBIGUOUS`, and the complement `NON_GIVEN_NAME_PARTICLES` works from), not written as a property, so it too holds by construction (decisions/vocabulary-model, 2026-10-25). A relation construction does not give (`∩ == ∅` with a set the sub-set cannot logically overlap; a sub-set member that is also a title is silently inert, title handling consuming it first) goes in `check_pool`, with a mutation in `tests/v2/test_vocab.py`'s table. Do **not** forbid `titles ∩ particles` — that overlap is intentional (`freiherr`, `st`). On the v1 side a sub-set new in 2.x, such as `honorific_tails`, `conjunctions_ambiguous`, `after_given_titles` or `suffix_words_ambiguous`, gets no `Constants` attribute and rides in `_snapshot()` as an intersection with its base, the shim roster under the 2.0 conventions listing each (the v1 ATTRIBUTE names did not move in #293, only the module constants did). A runtime `.add()` to the sub-set's `SetManager` does **not** propagate to the parent's (same as `first_name_titles`→`titles`), so a caller adding a brand-new word adds it to both. Two traps a gating set met (#752 phase 2i). Where a caller listing a word in the BASE set has always meant "read it as this everywhere" (the 1.4 recipe `c.titles.add("dean")`), the gated words compile OUT of the base rather than into both, or the gate overrides the caller's listing: `TITLES_AMBIGUOUS` is not a subset of `TITLES` for that reason, and a word in both reads as a title. And `_VOCAB_FIELDS` is read whole by two rosters in `_pipeline/_vocab.py`, the capitals shape's "unlisted" (`in_any_wordlist`) and the comma contrast's "claimed as non-name" (`_NON_NAME_FIELDS`), so a new field joins both by default; a field whose words may be names leaves both, which the grid found as `John Smith, DEAN` reversed where C6 had read a credential; #789's `given_name_abbreviations` met it again (`John Smith, JAS` and `Jas SMITH, XYZ` reversed), found by review because its grid held only the dotted spellings the set was for -- a new field's grid holds the spellings it must NOT move too. -**Validating a new parsing rule** — before implementing, simulate it in a throwaway script against `TEST_NAMES` (plus a few target-language examples) to catch regressions/false-positives early. E.g. this surfaced the `last_base` `do`/`st`/`mc` empties and the patronymic `"David Michael Abramovich"` false-reorder. After the change, `tests/v2/test_snapshot.py` lists every corpus name it moved, grouped by changed field -- read them, and re-record with `uv run python tools/differential/snapshot.py --record` once every move is intended, so the PR diff of `snapshot.jsonl` shows the reviewer the same list. `uv run python tools/differential/cldr_agreement.py` reports how the tree's agreement with CLDR's stated name fields moved; `tests/v2/test_cldr_agreement.py` fails on any move, and `tools/differential/cldr_agreement.json` doubles as a measured list of names the parser and CLDR read differently. The same command and test score the GND corpus's real names (`gnd_fields.jsonl`, record `gnd_agreement.json`) by the same rules, and `--record` rewrites every record, printing per source what it accepts. A locale-pack change reads the second view the same command prints, per pack (`cldr_agreement_locales.json`, #672, and `wikidata_native_agreement_locales.json` for real Han and kana names); its `JA` rows run only with the `ja` extra installed. A change to Han, Hangul or kana order or to Korean segmentation reads `wikidata_native_agreement.json` too: the only real names in those scripts the corpora hold. It holds no honorific, so a change to the honorific peel moves nothing there. A change to comma or credential handling also reads the local corpus of hand-typed names (`snapshot.py --local ctgov-officials --base `, which reads the base from a baseline kept for its package's files and never the shared snapshot beside the corpus (a base read once from the same fetch is copied rather than parsed again; `--sample N` reads a fixed sample while iterating; the shared snapshot beside the corpus is one file per source for every worktree, and #752's phase 2f found ctgov reporting a sibling branch's 1,334 joined-credential readings as its own moves against it); tools/differential/README.md's "Local corpora"): the one population of names nobody chose for a test or a catalog, so it shows how often each shape is typed, and the one no CI run sees. A change to title handling reads `ukec-individuals` the same way: UK donors' names in spoken order and mixed case, in 8,219 of whose 15,214 `parse()` read a title on 2026-10-22. A change to particles or generational words reads `snapshot.py --local tse-donors` the same way, Brazilian names in spoken order dense with both. A change to initials, to a title word outside English, or to how a script's combining marks are read reads `snapshot.py --local nepal-candidates` the same way: Nepal's local election candidates, each a romanized name (surnames written as initials, `B.K.`, `K.C.`; honorific titles such as `Guru` in front, `Shekh`, a title there until #752's phase 2j made it one only inside a title phrase, and `Md.`, which its period makes a title by shape) beside the same name in Devanagari, written with vowel signs and the virama, combining marks the letter-shape tests do not count as letters (decisions/nfd-shapes), and with the joiners ZWJ and ZWNJ. It is the only large population of South Asian names the corpora hold, and its corpus reads both scripts. A change to non-Latin scripts or to Arabic-script names reads `snapshot.py --local jrc-names` the same way, and `local_corpora.py groups jrc-names`: the spellings of one person across scripts and transliterations, harvested from news text, with 17,074 Arabic-script spellings where the committed corpora hold 315 distinct names with an Arabic letter, 96 of them CLDR's fictional renderings (2026-10-24), and under a licence that keeps even a reading of one of its names in the cache. A change to which field a word lands in also reads `local_agreement.py psc-individuals --base ` the same way: real names with the four boxes their filers typed them into (UK companies' people with significant control), scored field by field and by nationality; and `local_agreement.py orcid-researchers --base `, ORCID holders' own two boxes (given names, family name) from every country, scored by the country they work or live in -- a population each person split for themselves, academic but worldwide, the one to read for a change a language pack makes. These are the only real fielded populations outside the committed fields files, and no CI run sees them either. +**Validating a new parsing rule** — before implementing, simulate it in a throwaway script against `TEST_NAMES` (plus a few target-language examples) to catch regressions/false-positives early. E.g. this surfaced the `last_base` `do`/`st`/`mc` empties and the patronymic `"David Michael Abramovich"` false-reorder. After the change, `tests/v2/test_snapshot.py` lists every corpus name it moved, grouped by changed field -- read them, and re-record with `uv run python tools/differential/snapshot.py --record` once every move is intended, so the PR diff of `snapshot.jsonl` shows the reviewer the same list. `uv run python tools/differential/cldr_agreement.py` reports how the tree's agreement with CLDR's stated name fields moved; `tests/v2/test_cldr_agreement.py` fails on any move, and `tools/differential/cldr_agreement.json` doubles as a measured list of names the parser and CLDR read differently. The same command and test score the GND corpus's real names (`gnd_fields.jsonl`, record `gnd_agreement.json`) by the same rules, and `--record` rewrites every record, printing per source what it accepts. A locale-pack change reads the second view the same command prints, per pack (`cldr_agreement_locales.json`, #672, and `wikidata_native_agreement_locales.json` for real Han and kana names); its `JA` rows run only with the `ja` extra installed. A change to Han, Hangul or kana order or to Korean segmentation reads `wikidata_native_agreement.json` too: the only real names in those scripts the corpora hold. It holds no honorific, so a change to the honorific peel moves nothing there. A change to comma or credential handling also reads the local corpus of hand-typed names (`snapshot.py --local ctgov-officials --base `, which reads the base from a baseline kept for its package's files and never the shared snapshot beside the corpus (a base read once from the same fetch is copied rather than parsed again; `--sample N` reads a fixed sample while iterating; the shared snapshot beside the corpus is one file per source for every worktree, and #752's phase 2f found ctgov reporting a sibling branch's 1,334 joined-credential readings as its own moves against it); tools/differential/README.md's "Local corpora"): the one population of names nobody chose for a test or a catalog, so it shows how often each shape is typed, and the one no CI run sees. A change to title handling reads `ukec-individuals` the same way: UK donors' names in spoken order and mixed case, in 8,219 of whose 15,214 `parse()` read a title on 2026-10-22. A change to particles or generational words reads `snapshot.py --local tse-donors` the same way, Brazilian names in spoken order dense with both. A change to initials, to a title word outside English, or to how a script's combining marks are read reads `snapshot.py --local nepal-candidates` the same way: Nepal's local election candidates, each a romanized name (surnames written as initials, `B.K.`, `K.C.`; honorific titles such as `Guru` in front, `Shekh`, a title there until #752's phase 2j made it one only inside a title phrase, and `Md.`, which its period makes a title by shape) beside the same name in Devanagari, written with vowel signs and the virama, combining marks the letter-shape tests do not count as letters (decisions/nfd-shapes), and with the joiners ZWJ and ZWNJ. It is the only large population of South Asian names the corpora hold, and its corpus reads both scripts. A change to non-Latin scripts or to Arabic-script names reads `snapshot.py --local jrc-names` the same way, and `local_corpora.py groups jrc-names`: the spellings of one person across scripts and transliterations, harvested from news text, with 17,074 Arabic-script spellings where the committed corpora hold 315 distinct names with an Arabic letter, 96 of them CLDR's fictional renderings (2026-10-24), and under a licence that keeps even a reading of one of its names in the cache. A change to which field a word lands in also reads `local_agreement.py psc-individuals --base ` the same way: real names with the four boxes their filers typed them into (UK companies' people with significant control), scored field by field and by nationality; and `local_agreement.py orcid-researchers --base `, ORCID holders' own two boxes (given names, family name) from every country, scored by the country they work or live in -- a population each person split for themselves, academic but worldwide, the one to read for a change a language pack makes; and `local_agreement.py patentsview-inventors --base `, US patents' inventors in the two boxes their filings give, split by a professional filer, with how many people filed under each name in each country, the one to read for how common a name's word is among a country's US-patent inventors. These are the only real fielded populations outside the committed fields files, and no CI run sees them either. ## Gotchas @@ -305,7 +305,7 @@ The 2.0 rewrite lands as underscore-private modules alongside the v1 code. These **Perf regressions are caught by the scaling tests, not the absolute-time ones** — the `_thousand_names` tests use constant-size, delimiter-free input and are structurally blind to a complexity regression. A new inner loop, a policy-gated stage or a shape that needs a prefix gets a row in `tests/v2/test_benchmark.py`: its module docstring says which table, and the comment beside each table carries the measurements behind its rule. Coverage is collected on ONE interpreter, 3.14's (`COVERAGE_ON` in `.github/workflows/python-package.yml`, whose comment says why). -**A JSONL file written with `json.dumps(..., ensure_ascii=False)` is cut by `str.splitlines()`** — `json.dumps` leaves U+2028, U+2029 and U+0085 unescaped, and `splitlines()` splits at all three, so one name pasted from a document holding a line separator turns its line into two halves of invalid JSON, and the file into one no reader can load. #801's review found it in the local corpora (a single official among 536,981 would have made the cache unloadable for good), and the review of that fix found it again one file over, in `snapshot.py`'s `dump`/`load`, which the fix had not touched. So it is fixed at the shared source now: `corpus_build.jsonl_line` escapes the three and `corpus_build.jsonl_lines` cuts at `"\n"` only, and `corpus_build`'s corpus and fields writers, `snapshot.py`, `cldr_agreement.py`, `local_corpora.py`, `local_psc.py`'s extracts and `local_agreement.py` go through them. A new JSONL writer or reader in tools/ goes through the same two. Still outside them: the builders that write their own lines (`build_cjk_corpus.py`, `_conventions`, `_shapes`, `_rules`, `_issues`), `compare.py`'s worker stream, the tests' corpus reads (`_differential_fixtures.py`, `test_corpora.py`), and `local_orcid`'s extract, streamed through `gzip.open(..., newline="\n")` rather than read whole for its 25M lines, which cuts at `"\n"` only as `jsonl_lines` does; none of the 16 committed JSONL files held one on 2026-10-21. +**A JSONL file written with `json.dumps(..., ensure_ascii=False)` is cut by `str.splitlines()`** — `json.dumps` leaves U+2028, U+2029 and U+0085 unescaped, and `splitlines()` splits at all three, so one name pasted from a document holding a line separator turns its line into two halves of invalid JSON, and the file into one no reader can load. #801's review found it in the local corpora (a single official among 536,981 would have made the cache unloadable for good), and the review of that fix found it again one file over, in `snapshot.py`'s `dump`/`load`, which the fix had not touched. So it is fixed at the shared source now: `corpus_build.jsonl_line` escapes the three and `corpus_build.jsonl_lines` cuts at `"\n"` only, and `corpus_build`'s corpus and fields writers, `snapshot.py`, `cldr_agreement.py`, `local_corpora.py`, `local_psc.py`'s extracts and `local_agreement.py` go through them. A new JSONL writer or reader in tools/ goes through the same two. Still outside them: the builders that write their own lines (`build_cjk_corpus.py`, `_conventions`, `_shapes`, `_rules`, `_issues`), `compare.py`'s worker stream, the tests' corpus reads (`_differential_fixtures.py`, `test_corpora.py`), and the extracts of `local_orcid` and `local_patentsview`, streamed through `gzip.open(..., newline="\n")` rather than read whole for their 25M and 5.3M lines, which cuts at `"\n"` only as `jsonl_lines` does; none of the 16 committed JSONL files held one on 2026-10-21. **A row count does not prove a CSV whole: Python's `csv` returns a body cut inside a row as a short last row, without raising** -- even inside an open quote (`'a,b\r\n"x,y'` gives `[['a', 'b'], ['x,y']]`). So a fetch that checks its row count against the source's stated total, and tolerates a few malformed rows, passes a cut body one name short. #842's review found it in `ukec-individuals` (two reviewers independently); the fix requires the body to end with a line break and its last row not to be short (a cut only shortens a row; a long last row is left to the malformed share), with a test for each arm, since a quoted field holding a line break lets a cut end at one. A new CSV source (a `tools/differential/local_.py` module) takes the same check (`local_ukec._ukec_rows`'s last-row test), and its skip shares are tested at their shipped values, not only patched open for a small fixture. diff --git a/docs/design/mechanisms.md b/docs/design/mechanisms.md index ed9e7ecd1..5bbab07fd 100644 --- a/docs/design/mechanisms.md +++ b/docs/design/mechanisms.md @@ -426,7 +426,7 @@ print(len(MAIDEN_MARKERS), 'members ->', len(cls), 'structural classes')" **How it works.** WATCH asks whether a change moved a name: a committed corpus, `snapshot.jsonl` and a test that fails, or a local corpus read with `snapshot.py --local`, which no CI run sees. SCORE asks whether the parser agrees with someone who knows the fields: a committed fields file and agreement record, read by `cldr_agreement.py` from the repository alone, or a local corpus whose records carry the fields their filers, or the bearers themselves, typed, with its record kept in the cache beside it (`local_agreement.py`). The local score tells a change's author what moved among real filed names, by field, and gives agreement by group and how often each disagreement shape occurs. It cannot guard anything: no CI run and no reviewer sees it, and no test fails when it moves (the tool itself exits 1). Its truth is the filer's, so a box filled wrongly (a job typed as a title) reads as a miss unless the scorer sets it aside, and the scorer has to say what it sets aside; where the bearer typed the boxes, a miss is a disagreement with the person, though one who typed a family-first name into the given box reads as a miss too. COUNT asks how one word is used: a cached source behind a row of `tools/name_usage.py`, with only the counts cited. DERIVE asks what a shipped list should hold: a cached source and a script writing the vocabulary (`tools/zh_surnames.py`). SURVEY asks how people write whole names, and how OFTEN they write each shape: a cached local corpus (`tools/differential/local_corpora.py`). -A second axis cuts across the jobs: whether the names are real or invented, labelled with fields or bare text, chosen or unselected. Every committed corpus is CHOSEN: invented to fill fields (CLDR, the conventions table, the rules examples), cataloged (GND), filtered to names whose every word aligns with a stated part (both Wikidata corpora, a filter that drops credentials and typos), or picked by someone testing or reporting a case (v1's test names in `corpus.jsonl`, the case-table rows in `corpus_cjk.jsonl` and `corpus_shapes.jsonl`, and the strings reporters typed into `corpus_issues.jsonl`). The last group does hold hand-typed shapes — a job title in the name (`Bob Jones, author`), a credential before the comma (`Washington Jr. MD, Franklin`) — but one instance of each, as chosen. What SURVEY adds is the unselected population and its frequencies: ClinicalTrials.gov's study officials (local corpus `ctgov-officials`, measured 2026-10-21) are typed into a free-text field by sponsors' staff, so they show how often each shape occurs, and hold some no committed corpus does, such as `MBBS,FCPS` with no space. The FEC's individual contributors (local corpus `fec-individuals`, measured 2026-10-22) are unselected too, but mostly a filer's software writing a form's fields in a fixed order, so nearly all of its 1.89 million names share one shape (capitals, family, comma, given) and the hand-typed part is what goes inside the fields (a credential run or a suffix after a second comma): the population for that shape's variants, and blind to case. The UK Electoral Commission's individual donors (local corpus `ukec-individuals`, measured 2026-10-22) are the opposite shape: names as a donee's return gave them, mixed case and in spoken order, `parse()` reading a title in 8,219 of 15,214 (`Mr`, `Lord`, `Sir`) and only 2 holding a comma. Brazil's individual donors to candidates (local corpus `tse-donors`, measured 2026-10-22) are the FEC's comma-less counterpart: 848,351 filed names, nearly all in capitals and in spoken order, full of particles and agnomens (`DE`, `DA`, `DOS`; `FILHO`, `NETO`, `JUNIOR`), each beside the tax registry's name for the same donor, so a typed form can be read against its reference form (205,925 of 854,531 pairs differ). A source that pairs two spellings of one person scores without fields: the parse of each spelling should find the same surname, so where the spellings differ only in what cannot move a surname (accents, case), a disagreement is the parser reading one name two ways, found with no one labelling either (`local_corpora.py pairs --agreement`; on `tse-donors` on 2026-10-22, 1,304 of 43,552 accents-only pairs, 1,121 of them from `JÚNIOR`/`JUNIOR` and `SÁ`/`SA`). Read it per shape: a pair that leaves a word out can change the surname for real. JRC-Names (local corpus `jrc-names`, measured 2026-10-24) widens the pair to a group: one person's spellings harvested from news text across scripts, 218,822 persons with two or more. Words in two scripts cannot be compared, so `local_corpora.py groups` compares the readings' structure instead (title, suffix, given, family, the family's length and where it stands), counting persons. Read it per pair of scripts, since a structure can differ by convention: a script read family first stands its family elsewhere, and a writing variant moves the reading for real (the Arabic comma, katakana with and without `・`). Companies House's people with significant control (local corpus `psc-individuals`, measured 2026-10-22) are unselected and labelled at once: each filer types the name into four boxes (title, forename, middle name, surname) and states a nationality, so the corpus is the one real population a local SCORE reads by nationality. It is filed through a form's boxes, so it holds almost no comma (838 of 1,040,708 sampled records) or credential. ORCID's researchers (local corpus `orcid-researchers`, measured 2026-10-27) are labelled by the bearer rather than a filer: each holder typed their own name into two boxes (given names, family name), from 250 countries. Two things keep it from being a SURVEY. It keeps at most 20,000 records a country, so it shows which shapes each country holds and how the parser reads them, never how often a shape occurs across countries. And its country is where a holder works or lives, most often taken from an employment, so a country's row is its institutions' researchers, not its nationals. It is the SCORE to read for a language pack: 80.5% of its 1,831,533 records agreed on every field, as few as 13.4% in Peru, and the other 17 countries of at least 1,000 records under half were Brazil and 16 Spanish-speaking ones (Argentina read 80.3%); in the two commonest disagreement shapes, 249,526 of the 357,517 records that disagreed, a two-word family box was read as one word (measured on master at 598167f4). One source may do several jobs, each through its own code: Wikidata does COUNT live and WATCH and SCORE through its builders. A live source has no fixed file to pin, so its cached copy is the pin, carrying the source's own data date and a sha256 checked on every load (a truncated file otherwise reads as a run of false zeros, #775). +A second axis cuts across the jobs: whether the names are real or invented, labelled with fields or bare text, chosen or unselected. Every committed corpus is CHOSEN: invented to fill fields (CLDR, the conventions table, the rules examples), cataloged (GND), filtered to names whose every word aligns with a stated part (both Wikidata corpora, a filter that drops credentials and typos), or picked by someone testing or reporting a case (v1's test names in `corpus.jsonl`, the case-table rows in `corpus_cjk.jsonl` and `corpus_shapes.jsonl`, and the strings reporters typed into `corpus_issues.jsonl`). The last group does hold hand-typed shapes — a job title in the name (`Bob Jones, author`), a credential before the comma (`Washington Jr. MD, Franklin`) — but one instance of each, as chosen. What SURVEY adds is the unselected population and its frequencies: ClinicalTrials.gov's study officials (local corpus `ctgov-officials`, measured 2026-10-21) are typed into a free-text field by sponsors' staff, so they show how often each shape occurs, and hold some no committed corpus does, such as `MBBS,FCPS` with no space. The FEC's individual contributors (local corpus `fec-individuals`, measured 2026-10-22) are unselected too, but mostly a filer's software writing a form's fields in a fixed order, so nearly all of its 1.89 million names share one shape (capitals, family, comma, given) and the hand-typed part is what goes inside the fields (a credential run or a suffix after a second comma): the population for that shape's variants, and blind to case. The UK Electoral Commission's individual donors (local corpus `ukec-individuals`, measured 2026-10-22) are the opposite shape: names as a donee's return gave them, mixed case and in spoken order, `parse()` reading a title in 8,219 of 15,214 (`Mr`, `Lord`, `Sir`) and only 2 holding a comma. Brazil's individual donors to candidates (local corpus `tse-donors`, measured 2026-10-22) are the FEC's comma-less counterpart: 848,351 filed names, nearly all in capitals and in spoken order, full of particles and agnomens (`DE`, `DA`, `DOS`; `FILHO`, `NETO`, `JUNIOR`), each beside the tax registry's name for the same donor, so a typed form can be read against its reference form (205,925 of 854,531 pairs differ). A source that pairs two spellings of one person scores without fields: the parse of each spelling should find the same surname, so where the spellings differ only in what cannot move a surname (accents, case), a disagreement is the parser reading one name two ways, found with no one labelling either (`local_corpora.py pairs --agreement`; on `tse-donors` on 2026-10-22, 1,304 of 43,552 accents-only pairs, 1,121 of them from `JÚNIOR`/`JUNIOR` and `SÁ`/`SA`). Read it per shape: a pair that leaves a word out can change the surname for real. JRC-Names (local corpus `jrc-names`, measured 2026-10-24) widens the pair to a group: one person's spellings harvested from news text across scripts, 218,822 persons with two or more. Words in two scripts cannot be compared, so `local_corpora.py groups` compares the readings' structure instead (title, suffix, given, family, the family's length and where it stands), counting persons. Read it per pair of scripts, since a structure can differ by convention: a script read family first stands its family elsewhere, and a writing variant moves the reading for real (the Arabic comma, katakana with and without `・`). Companies House's people with significant control (local corpus `psc-individuals`, measured 2026-10-22) are unselected and labelled at once: each filer types the name into four boxes (title, forename, middle name, surname) and states a nationality, so the corpus is the one real population a local SCORE reads by nationality. It is filed through a form's boxes, so it holds almost no comma (838 of 1,040,708 sampled records) or credential. ORCID's researchers (local corpus `orcid-researchers`, measured 2026-10-27) are labelled by the bearer rather than a filer: each holder typed their own name into two boxes (given names, family name), from 250 countries. Two things keep it from being a SURVEY. It keeps at most 20,000 records a country, so it shows which shapes each country holds and how the parser reads them, never how often a shape occurs across countries. And its country is where a holder works or lives, most often taken from an employment, so a country's row is its institutions' researchers, not its nationals. It is the SCORE to read for a language pack: 80.5% of its 1,831,533 records agreed on every field, as few as 13.4% in Peru, and the other 17 countries of at least 1,000 records under half were Brazil and 16 Spanish-speaking ones (Argentina read 80.3%); in the two commonest disagreement shapes, 249,526 of the 357,517 records that disagreed, a two-word family box was read as one word (measured on master at 598167f4). PatentsView's inventors (local corpus `patentsview-inventors`, measured 2026-10-28) are labelled by a professional filer, the applicant's attorney, and carry the number of people who filed under each name in each country, kept uncapped in the extract: the one fielded source that counts the people under each name, so a word's frequency among a country's US-patent inventors (not its population) can be read from it. Its agreement reads beside ORCID's as how a name is written against how its bearer types it: 99.8% in Japan, 99.5% in China and 99.4% in Korea, where ORCID read 95.5%, 84.5% and 92.4%, while Spain reads about as low in both (45.6% here, 48.0% in ORCID). One source may do several jobs, each through its own code: Wikidata does COUNT live and WATCH and SCORE through its builders. A live source has no fixed file to pin, so its cached copy is the pin, carrying the source's own data date and a sha256 checked on every load (a truncated file otherwise reads as a run of false zeros, #775). **Reach for it when.** A source is proposed, before choosing between a builder and a tool column; and when a corpus would hold real people's names nobody vetted for publication, which is the case for a local corpus rather than a committed one. diff --git a/tests/v2/test_local_agreement.py b/tests/v2/test_local_agreement.py index 51cf95b17..0d9484b51 100644 --- a/tests/v2/test_local_agreement.py +++ b/tests/v2/test_local_agreement.py @@ -266,26 +266,35 @@ def test_a_record_written_by_another_scorer_is_refused( monkeypatch: pytest.MonkeyPatch) -> None: _corpus(monkeypatch, *_ALL) assert agreement.main([_PSC, "--record"]) == 0 - before = agreement.scorer_fingerprint() + before = agreement.scorer_fingerprint(_PSC) # Each part of the code that decides a record's entry moves the # fingerprint: how a field is compared, which records get an entry, - # and how an entry is keyed. + # how an entry is keyed, and the source's own boxes; another source's + # boxes do not, so adding a source leaves this one's record standing. + other = next(s for s in agreement.FIELDED if s != _PSC) + with monkeypatch.context() as patch: + patch.setitem(agreement.FIELDED, other, + agreement.Fielded({"forename": "given", "surname": "family"}, group="x")) + assert agreement.scorer_fingerprint(_PSC) == before, other + patch.setitem(agreement.FIELDED, _PSC, + agreement.Fielded({"forename": "given", "surname": "family"}, group="x")) + assert agreement.scorer_fingerprint(_PSC) != before, "its own boxes" for name, changed in (("fold", lambda text: text.casefold()), ("score", lambda rows, fielded, parser: None), ("key", lambda record: str(record))): with monkeypatch.context() as patch: patch.setattr(agreement, name, changed) - assert agreement.scorer_fingerprint() != before, name + assert agreement.scorer_fingerprint(_PSC) != before, name with monkeypatch.context() as patch: patch.setattr(agreement.corpus_build, "jsonl_line", lambda obj, sort_keys=False: str(obj)) - assert agreement.scorer_fingerprint() != before, "jsonl_line" + assert agreement.scorer_fingerprint(_PSC) != before, "jsonl_line" # fold() casefolds, so a Python whose Unicode database moves it reads # names differently; its moves would be the database's, not the parser's. with monkeypatch.context() as patch: patch.setattr(unicodedata, "unidata_version", "0.0.0") - assert agreement.scorer_fingerprint() != before, "unidata_version" - assert agreement.scorer_fingerprint() == before + assert agreement.scorer_fingerprint(_PSC) != before, "unidata_version" + assert agreement.scorer_fingerprint(_PSC) == before monkeypatch.setattr(agreement, "fold", lambda text: text.casefold()) with pytest.raises(SystemExit, match="another version of this scorer"): agreement.main([_PSC]) diff --git a/tests/v2/test_local_corpora.py b/tests/v2/test_local_corpora.py index d5125351d..d895c251d 100644 --- a/tests/v2/test_local_corpora.py +++ b/tests/v2/test_local_corpora.py @@ -49,9 +49,10 @@ # the fetches read, so a patch reaches the code only when it is set on # the module holding the name (local_core.RETRIES, local_fec.FEC_MAX_BAD). # A second load by path (load_tool) would be a copy nothing reads. -core, ctgov, fec, tse, tw, ukec, psc, jrc, nepal, orcid = ( +core, ctgov, fec, tse, tw, ukec, psc, jrc, nepal, orcid, patentsview = ( importlib.import_module(f"local_{key}") - for key in ("core", "ctgov", "fec", "tse", "tw", "ukec", "psc", "jrc", "nepal", "orcid")) + for key in ("core", "ctgov", "fec", "tse", "tw", "ukec", "psc", "jrc", "nepal", "orcid", + "patentsview")) _SOURCE = "ctgov-officials" _DATE = "2026-10-21T09:00:05" @@ -3346,10 +3347,11 @@ def test_no_module_imports_a_sibling_constant() -> None: # module. A `from local_x import CONSTANT` would bind a second name a # patch on local_x cannot reach, and a test could pass with its guard # off. The modules are SOURCES' own, so a new source is covered. PSC, - # JRC and ORCID are sources' names, keys of SOURCES, and no fetch's setting. + # JRC, ORCID and PV are sources' names, keys of SOURCES, and no fetch's setting. import ast sources = {source.fetch.__module__ for source in local.SOURCES.values()} - assert sources == {m.__name__ for m in (ctgov, fec, tse, tw, ukec, psc, jrc, nepal, orcid)} + assert sources == {m.__name__ for m in (ctgov, fec, tse, tw, ukec, psc, jrc, nepal, orcid, + patentsview)} siblings = {"local_corpora", "local_core", *sources} imported = {(path.stem, alias.name) for path in (Path(str(local.__file__)).parent / f"{s}.py" @@ -3358,7 +3360,7 @@ def test_no_module_imports_a_sibling_constant() -> None: if isinstance(node, ast.ImportFrom) and node.module in siblings for alias in node.names if alias.name.lstrip("_").isupper()} assert imported == {("local_corpora", "PSC"), ("local_corpora", "JRC"), - ("local_corpora", "ORCID")} + ("local_corpora", "ORCID"), ("local_corpora", "PV")} def test_the_fold_lists_everything_its_functions_call_and_read() -> None: diff --git a/tests/v2/test_local_patentsview.py b/tests/v2/test_local_patentsview.py new file mode 100644 index 000000000..f05741f5b --- /dev/null +++ b/tests/v2/test_local_patentsview.py @@ -0,0 +1,366 @@ +"""Offline checks on tools/differential/local_patentsview.py, the +patentsview-inventors local corpus. + +Every name here is invented, and no reading of a real inventor may +reach the repository: the inventors are living people, which is why +the corpus is local. No test reaches the portal or needs its key: the +server is a stub, and the key is a file in the test's own home. What +these pin is where the key goes, the refusal of another release or of +bytes that are not the pinned ones, the join of the two inventor tables +and its people count, the checks a read makes, and the cap. +""" +import csv +import email.message +import gzip +import hashlib +import importlib +import io +import json +import urllib.error +import urllib.request +import zipfile +from collections.abc import Callable, Sequence +from pathlib import Path + +import pytest + +from ._differential_fixtures import load_tool, own_cache + +local = load_tool("local_corpora") +# The module the fetch reads its constants from, by name (test_local_corpora.py). +pv = importlib.import_module("local_patentsview") + +_PV = "patentsview-inventors" + + +@pytest.fixture(autouse=True) +def _own_cache(monkeypatch: pytest.MonkeyPatch, tmp_path_factory: pytest.TempPathFactory) -> Path: + return own_cache(monkeypatch, tmp_path_factory) + + +@pytest.fixture +def key(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> Path: + path = tmp_path / "uspto-odp-key" + path.write_text("test-key-not-real\n") + monkeypatch.setattr(pv, "PV_KEY", path) + return path + + +def _table(header: list[str], rows: list[list[str]]) -> bytes: + """A table as PatentsView ships one: a zip of one TSV, every field quoted.""" + text = io.StringIO() + writer = csv.writer(text, delimiter="\t", quoting=csv.QUOTE_ALL, lineterminator="\n") + writer.writerow(header) + writer.writerows(rows) + raw = io.BytesIO() + with zipfile.ZipFile(raw, "w") as archive: + archive.writestr("table.tsv", text.getvalue()) + return raw.getvalue() + + +_RAW_HEAD = ["patent_id", "inventor_sequence", "inventor_id", "raw_inventor_name_first", + "raw_inventor_name_last", "deceased_flag", "rawlocation_id"] +_DIS_HEAD = ["patent_id", "inventor_sequence", "inventor_id", "disambig_inventor_name_first", + "disambig_inventor_name_last", "gender_code", "location_id"] +_LOC_HEAD = ["location_id", "disambig_city", "disambig_state", "disambig_country", + "latitude", "longitude", "county", "state_fips", "county_fips"] + +#: (patent, sequence, inventor id, first, last, location). Ann Lee holds +#: two patents (one person), a second Ann Lee is another person, and Bo +#: Wu files in Japan, Cy Doe's location names no country, and Eve Ng +#: holds four patents in Japan: 9 of the 10 rows name a country, which +#: is PV_MIN_COUNTRY's shipped floor exactly. +_INVENTORS = [("1", "0", "i1", "Ann", "Lee", "l1"), ("2", "0", "i1", "Ann", "Lee", "l1"), + ("2", "1", "i2", "Ann", "Lee", "l1"), ("3", "0", "i3", "Bo", "Wu", "l2"), + ("4", "0", "i4", "Cy Q.", "Doe", "l3"), ("5", "0", "i5", "Di", "Fox", "l2"), + *((str(n), "0", "i6", "Eve", "Ng", "l2") for n in range(6, 10))] +_LOCATIONS = [["l1", "X", "", "US", "0", "0", "", "", ""], + ["l2", "Y", "", "JP", "0", "0", "", "", ""], + ["l3", "Z", "", "", "0", "0", "", "", ""]] + + +def _files(inventors: Sequence[tuple[str, ...]] = _INVENTORS, *, + dis: Sequence[tuple[str, ...]] | None = None, + raw_extra: Sequence[list[str]] = ()) -> dict[str, bytes]: + raw_rows = [[p, s, i, f, last, "False", "r"] for p, s, i, f, last, _ in inventors] + dis_rows = [[p, s, i, f, last, "U", loc] for p, s, i, f, last, loc in (dis or inventors)] + return {pv.PV_RAW: _table(_RAW_HEAD, raw_rows + list(raw_extra)), + pv.PV_INVENTORS: _table(_DIS_HEAD, dis_rows), + pv.PV_LOCATIONS: _table(_LOC_HEAD, _LOCATIONS)} + + +class _Body(io.BytesIO): + def __enter__(self) -> "_Body": + return self + + +def _serve(monkeypatch: pytest.MonkeyPatch, files: dict[str, bytes], *, + listed: dict[str, tuple[int, str]] | None = None, rows: int | None = None) -> list[str]: + """`files` as the pinned release, their sizes and sha256 pinned and + the rows the raw table holds; the URLs asked.""" + monkeypatch.setattr(pv, "PV_FILES", { + name: (len(body), "2026-03-12 22:26:59", hashlib.sha256(body).hexdigest()) + for name, body in files.items()}) + monkeypatch.setattr(pv, "PV_ROWS", rows if rows is not None else len(_INVENTORS)) + asked: list[str] = [] + stated = listed or {name: (len(body), "2026-03-12 22:26:59") for name, body in files.items()} + + def opened(url: str) -> _Body: + asked.append(url) + if "includeFiles" in url: + return _Body(json.dumps({"bulkDataProductBag": [{"productFileBag": {"fileDataBag": [ + {"fileName": n, "fileSize": size, "fileLastModifiedDateTime": when, + "fileDownloadURI": f"{pv.PV_API}/files/{pv.PV_PRODUCT}/{n}"} + for n, (size, when) in stated.items()]}}]}).encode()) + return _Body(files[url.rsplit("/", 1)[-1]]) + + monkeypatch.setattr(pv, "_open", opened) + return asked + + +# -- the key ------------------------------------------------------------------ + +@pytest.mark.parametrize("held", [None, "", "two words\n"]) +def test_a_missing_or_malformed_key_is_refused_saying_how_to_get_one( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path, held: str | None) -> None: + path = tmp_path / "uspto-odp-key" + if held is not None: + path.write_text(held) + monkeypatch.setattr(pv, "PV_KEY", path) + with pytest.raises(ValueError, match="ID.me.*My ODP.*chmod 600"): + pv._key() + + +def test_the_key_goes_to_an_https_uspto_host_only( + monkeypatch: pytest.MonkeyPatch, key: Path) -> None: + sent: list[urllib.request.Request] = [] + + class Opener: + def open(self, request: urllib.request.Request, timeout: int) -> None: + sent.append(request) + + monkeypatch.setattr(pv.urllib.request, "build_opener", lambda *handlers: Opener()) + pv._open(f"{pv.PV_API}/{pv.PV_PRODUCT}") + assert sent[0].get_header("X-api-key") == "test-key-not-real" + for elsewhere in ("http://api.uspto.gov/x", "https://uspto.gov.example.com/x", + "https://example.com/x"): + with pytest.raises(ValueError, match="the key goes nowhere else"): + pv._open(elsewhere) + assert len(sent) == 1 + + +@pytest.mark.parametrize(("to", "keeps"), [ + ("https://data.uspto.gov/file.zip", True), + ("https://bucket.s3.amazonaws.com/file.zip", False)]) +def test_a_redirect_off_uspto_drops_the_key(to: str, keeps: bool) -> None: + request = urllib.request.Request("https://api.uspto.gov/f", headers={"X-API-KEY": "k"}) + new = pv._KeyStaysHome().redirect_request(request, None, 302, "Found", {}, to) + assert new is not None and (new.get_header("X-api-key") == "k") is keeps + + +# -- the release -------------------------------------------------------------- + +@pytest.mark.parametrize("listed", [ + {pv.PV_RAW: (1, "2026-03-12 22:26:59")}, + "size", "date"]) +def test_a_listing_of_another_release_is_refused( + monkeypatch: pytest.MonkeyPatch, key: Path, + listed: dict[str, tuple[int, str]] | str) -> None: + files = _files() + if listed == "size": + listed = {n: (len(b) + 1, "2026-03-12 22:26:59") for n, b in files.items()} + elif listed == "date": + listed = {n: (len(b), "2026-06-11 00:00:00") for n, b in files.items()} + assert isinstance(listed, dict) + _serve(monkeypatch, files, listed=listed) + with pytest.raises(ValueError, match="another release, measured and pinned by hand"): + pv.renew_pv() + + +def test_bytes_that_are_not_the_pinned_sha256_are_refused_and_not_kept( + monkeypatch: pytest.MonkeyPatch, key: Path) -> None: + files = _files() + _serve(monkeypatch, files) + name = pv.PV_LOCATIONS + size, when, _ = pv.PV_FILES[name] + monkeypatch.setitem(pv.PV_FILES, name, (size, when, "0" * 64)) + with pytest.raises(ValueError, match="sha256"): + pv.renew_pv() + assert not pv.extract_path().exists() + # The files that arrived whole stay for the next run; nothing partial does. + assert sorted(p.name for p in pv.extract_path().parent.rglob("*") if p.is_file()) == [ + pv.PV_INVENTORS, pv.PV_RAW] + + +def test_a_run_after_a_failed_one_downloads_only_what_was_not_whole( + monkeypatch: pytest.MonkeyPatch, key: Path) -> None: + files = _files() + asked = _serve(monkeypatch, files) + size, when, sha = pv.PV_FILES[pv.PV_LOCATIONS] + monkeypatch.setitem(pv.PV_FILES, pv.PV_LOCATIONS, (size, when, "0" * 64)) + with pytest.raises(ValueError, match="sha256"): + pv.renew_pv() + monkeypatch.setitem(pv.PV_FILES, pv.PV_LOCATIONS, (size, when, sha)) + asked.clear() + pv.renew_pv() + assert [url.rsplit("/", 1)[-1] for url in asked if "includeFiles" not in url] == [ + pv.PV_LOCATIONS] + assert not (pv.extract_path().parent / "downloads").exists() + + +def test_a_download_dropped_mid_file_is_tried_again( + monkeypatch: pytest.MonkeyPatch, key: Path) -> None: + files = _files() + _serve(monkeypatch, files) + served: Callable[[str], _Body] = pv._open + drops = [pv.PV_RAW] + monkeypatch.setattr(importlib.import_module("local_core").time, "sleep", lambda _s: None) + + def dropping(url: str) -> _Body: + if drops and url.endswith(drops[0]): + drops.pop() + raise ConnectionResetError("reset by peer") + return served(url) + + monkeypatch.setattr(pv, "_open", dropping) + pv.renew_pv() + assert pv.extract_path().exists() and not drops + + +@pytest.mark.parametrize("code", [401, 403]) +def test_a_key_the_portal_refuses_is_reported_as_the_keys( + monkeypatch: pytest.MonkeyPatch, key: Path, code: int) -> None: + class Opener: + def open(self, request: urllib.request.Request, timeout: int) -> None: + raise urllib.error.HTTPError(request.full_url, code, "Unauthorized", + email.message.Message(), io.BytesIO()) + + monkeypatch.setattr(pv.urllib.request, "build_opener", lambda *handlers: Opener()) + with pytest.raises(ValueError, match=f"refused the key in .*uspto-odp-key \\({code}\\).*ID.me"): + pv._listed() + + +# -- the fetch ------------------------------------------------------------------ + +def test_a_fetch_counts_people_not_patents_and_keeps_the_country( + monkeypatch: pytest.MonkeyPatch, key: Path) -> None: + _serve(monkeypatch, _files()) + corpus = local.fetch(_PV) + assert sorted(corpus.records, key=lambda r: r["name"]) == [ + {"country": "US", "family": "Lee", "given": "Ann", "name": "Ann Lee", "people": "2"}, + {"country": "JP", "family": "Wu", "given": "Bo", "name": "Bo Wu", "people": "1"}, + {"family": "Doe", "given": "Cy Q.", "name": "Cy Q. Doe", "people": "1"}, + {"country": "JP", "family": "Fox", "given": "Di", "name": "Di Fox", "people": "1"}, + {"country": "JP", "family": "Ng", "given": "Eve", "name": "Eve Ng", "people": "1"}] + detail = corpus.meta["detail"] + assert (detail["rows"], detail["names"], detail["people_by_name"], + detail["rows_with_a_country"], detail["countries"]) == (10, 5, 6, 9, 2) + # The zips are gone; only the extract and the corpus files stay. + assert not (pv.extract_path().parent / "downloads").exists() + + +def test_the_extract_keeps_every_name_with_its_people_as_a_number( + monkeypatch: pytest.MonkeyPatch, key: Path) -> None: + _serve(monkeypatch, _files()) + local.fetch(_PV) + with gzip.open(pv.extract_path(), "rt", encoding="utf-8") as lines: + header, *names = lines.read().splitlines() + assert json.loads(header)["release"] == pv.PV_RELEASE + assert json.loads(names[0]) == {"country": "US", "family": "Lee", "given": "Ann", + "people": 2} + + +def test_the_two_inventor_tables_out_of_step_are_refused( + monkeypatch: pytest.MonkeyPatch, key: Path) -> None: + shuffled = _INVENTORS[2:] + _INVENTORS[:2] + _serve(monkeypatch, _files(dis=shuffled)) + with pytest.raises(ValueError, match="not row for row the same inventors"): + pv.renew_pv() + _serve(monkeypatch, _files(dis=_INVENTORS + [("9", "0", "i9", "Ed", "Ng", "l1")])) + with pytest.raises(ValueError, match="holds more rows than"): + pv.renew_pv() + + +def test_a_table_read_in_part_is_refused_and_the_extract_kept( + monkeypatch: pytest.MonkeyPatch, key: Path) -> None: + _serve(monkeypatch, _files()) + pv.renew_pv() + before = pv.extract_path().read_bytes() + _serve(monkeypatch, _files(), rows=len(_INVENTORS) + 1) + with pytest.raises(ValueError, match="a table was read in part"): + pv.renew_pv() + assert pv.extract_path().read_bytes() == before + # Nothing staged or partial stays beside it (the whole zips may, for the next run). + assert sorted(p.name for p in pv.extract_path().parent.iterdir() if p.is_file()) == [ + pv.extract_path().name] + + +def test_a_row_not_the_headers_width_is_counted_and_past_the_share_refused( + monkeypatch: pytest.MonkeyPatch, key: Path) -> None: + files = _files(raw_extra=[["6", "0", "i6", "Ed"]]) + _serve(monkeypatch, files) + with pytest.raises(ValueError, match="1 of 10 g_inventor_not_disambiguated.tsv.zip: " + "rows not 7 fields"): + pv.renew_pv() + monkeypatch.setattr(pv, "PV_MAX_BAD", 1 / 5) + assert local.fetch(_PV).meta["detail"][f"{pv.PV_RAW}: rows not 7 fields"] == 1 + + +@pytest.mark.parametrize(("inventors", "says"), [ + ([(p, s, i, f, last, "l3") for p, s, i, f, last, _ in _INVENTORS], + "only 0 of 10 rows name a country"), + ([(p, s, i, f, "", loc) for p, s, i, f, _, loc in _INVENTORS], + "only 0 of 10 rows fill both boxes")]) +def test_tables_whose_columns_moved_under_the_reader_are_refused( + monkeypatch: pytest.MonkeyPatch, key: Path, inventors: Sequence[tuple[str, ...]], + says: str) -> None: + _serve(monkeypatch, _files(inventors)) + with pytest.raises(ValueError, match=says): + pv.renew_pv() + + +def test_a_header_without_a_column_read_is_refused( + monkeypatch: pytest.MonkeyPatch, key: Path) -> None: + files = _files() + files[pv.PV_LOCATIONS] = _table(["location_id", "country_name"], [["l1", "US"]]) + _serve(monkeypatch, files) + with pytest.raises(ValueError, match="header holds no disambig_country"): + pv.renew_pv() + + +def test_a_file_that_is_not_an_extract_is_refused_naming_the_refresh() -> None: + path = pv.extract_path() + path.parent.mkdir(parents=True, exist_ok=True) + path.write_bytes(gzip.compress(b'{"release": "2024-12-31", "files": {}, "counts": {}}\n', + mtime=0)) + with pytest.raises(ValueError, match="not an extract this tool wrote.*--refresh"): + pv._pv_read(path) + + +def test_a_corpus_kept_under_another_cap_is_refused_on_load( + monkeypatch: pytest.MonkeyPatch, key: Path) -> None: + _serve(monkeypatch, _files()) + local.fetch(_PV) + assert local.load(_PV) is not None + monkeypatch.setattr(pv, "PV_CAP", 7) + with pytest.raises(local.CorpusDamaged, match="at most 7 names a country"): + local.load(_PV) + + +def test_the_cap_samples_a_large_country_and_keeps_the_same_names_whatever_the_order() -> None: + people = [{"name": f"Ann Lee{i}", "country": "US", "people": "1"} for i in range(30)] + kept = pv.pv_kept(iter(people + [{"name": "Bo Wu", "country": "JP", "people": "1"}]), 10) + assert sum(r["country"] == "US" for r in kept) == 10 + again = pv.pv_kept(iter(people[::-1]), 10) + assert sorted(map(str, again)) == sorted(map(str, (r for r in kept if r["country"] == "US"))) + + +def test_the_corpus_scores_with_its_boxes(monkeypatch: pytest.MonkeyPatch, key: Path) -> None: + agreement = load_tool("local_agreement") + _serve(monkeypatch, _files()) + corpus = local.fetch(_PV) + from nameparser import Parser + parser = Parser() + listed = agreement._title_words(parser.lexicon) + fielded = agreement.FIELDED[_PV] + assert all(not agreement.read(r, fielded, parser, listed).got for r in corpus.records) diff --git a/tests/v2/test_measure.py b/tests/v2/test_measure.py index 5662e0ec9..92a1b0759 100644 --- a/tests/v2/test_measure.py +++ b/tests/v2/test_measure.py @@ -34,13 +34,14 @@ def _fetch(cache: Path, *sources: str) -> None: def test_the_plan_runs_each_check_for_the_corpora_fetched(_own_cache: Path, tmp_path: Path) -> None: _fetch(_own_cache, "ctgov-officials", corpora.PSC, corpora.JRC, corpora.ORCID, - "tse-donors") + corpora.PV, "tse-donors") steps = measure.plan(tmp_path, 3) assert [s.label for s in steps] == [ "committed snapshot", "CLDR and GND agreement", "local snapshot ctgov-officials", f"local snapshot {corpora.JRC}", - f"local snapshot {corpora.ORCID}", f"local snapshot {corpora.PSC}", - "local snapshot tse-donors", f"agreement {corpora.ORCID}", + f"local snapshot {corpora.ORCID}", f"local snapshot {corpora.PV}", + f"local snapshot {corpora.PSC}", "local snapshot tse-donors", + f"agreement {corpora.ORCID}", f"agreement {corpora.PV}", f"agreement {corpora.PSC}", f"alike {corpora.JRC}", "alike tse-donors"] for step in steps: # Every step through the wrapper: unwrapped, a refusal reads as moves. @@ -52,7 +53,7 @@ def test_the_plan_runs_each_check_for_the_corpora_fetched(_own_cache: Path, assert step.against == measure.AGAINST_RECORD assert steps[0].command[-2:] == ["--jobs", "3"] assert measure.unmeasured() == sorted(set(corpora.SOURCES) - { - "ctgov-officials", corpora.PSC, corpora.JRC, corpora.ORCID, "tse-donors"}) + "ctgov-officials", corpora.PSC, corpora.JRC, corpora.ORCID, corpora.PV, "tse-donors"}) def test_the_plan_with_nothing_fetched_runs_the_committed_checks(tmp_path: Path) -> None: diff --git a/tools/differential/README.md b/tools/differential/README.md index 7696b868b..62dcdbc76 100644 --- a/tools/differential/README.md +++ b/tools/differential/README.md @@ -248,7 +248,7 @@ uv run python tools/differential/build_shapes_corpus.py --coverage A local corpus holds real names that are not committed: fetched into the cache by `local_corpora.py`, read by `snapshot.py --local` and by a row of `tools/name_usage.py`, scored by `local_agreement.py` where its records carry their fields, and seen by no reviewer and no CI run (the SURVEY job in mechanisms.md#NAME-DATA-JOBS, and WATCH and SCORE outside CI). What reaches the repository is what a contributor cites from it, counts and shapes written with made-up names. -`local_corpora.py` holds the `SOURCES` registry, the cache's files, the views and the command line; each source's fetch is a module of its own (`local_ctgov.py`, `local_fec.py`, `local_tse.py`, `local_tw.py`, `local_ukec.py`, `local_psc.py`, `local_jrc.py`, `local_nepal.py`, `local_orcid.py`) whose docstring says what the source is and what its terms allow, built on the retry policy, share gate and downloads in `local_core.py`. A new source is a module and a `SOURCES` row. A source whose records hold a value its own code derives, rather than one the source gave (`jrc-names`' script label), stores a read version in `meta.json`'s detail and refuses on load a corpus read by another (`JRC_READ_VERSION`, bumped by hand with any change to how a record is built; `psc-individuals` does the same for its cached extracts, with `PSC_EXTRACT_VERSION` in each extract's file name): the sha256 checks prove the files are the ones fetched, not that today's code would write them. A constant such as `TSE_MAX_BAD` or `RETRIES` lives in one module, which every reader reads it through, so a test patches it there; no other module `from`-imports it, so a patch set elsewhere raises rather than missing the code (`test_no_module_imports_a_sibling_constant` holds this). +`local_corpora.py` holds the `SOURCES` registry, the cache's files, the views and the command line; each source's fetch is a module of its own (`local_ctgov.py`, `local_fec.py`, `local_tse.py`, `local_tw.py`, `local_ukec.py`, `local_psc.py`, `local_jrc.py`, `local_nepal.py`, `local_orcid.py`, `local_patentsview.py`) whose docstring says what the source is and what its terms allow, built on the retry policy, share gate and downloads in `local_core.py`. A new source is a module and a `SOURCES` row. A source whose records hold a value its own code derives, rather than one the source gave (`jrc-names`' script label), stores a read version in `meta.json`'s detail and refuses on load a corpus read by another (`JRC_READ_VERSION`, bumped by hand with any change to how a record is built; `psc-individuals` does the same for its cached extracts, with `PSC_EXTRACT_VERSION` in each extract's file name): the sha256 checks prove the files are the ones fetched, not that today's code would write them. A constant such as `TSE_MAX_BAD` or `RETRIES` lives in one module, which every reader reads it through, so a test patches it there; no other module `from`-imports it, so a patch set elsewhere raises rather than missing the code (`test_no_module_imports_a_sibling_constant` holds this). ``` uv run python tools/differential/local_corpora.py fetch ctgov-officials @@ -284,6 +284,7 @@ Any snapshot reading of 5,000 entries or more, committed or local, parses in a w | `ukec-individuals` | the distinct names of the individual donors in the UK Electoral Commission's register of donations (to parties, their members, third parties, permitted participants and regulated donees), from the search's CSV export under the site's own default filter, fetched whole on every fetch (9.5 MB on 2026-10-22, accepted dates from 2001 to 2026). The default leaves out donations reported only before a poll (`prePoll=false`: 374 rows and 21 names on 2026-10-22). A name as the donee's return gave it, mostly in its spoken order and mixed case, with the donor's title: of its 15,214 names on 2026-10-22, `parse()` read a title in 8,219 (its `title` field non-empty) (`Mr`, `Ms`, `Mrs`, `Lord`, `Dr`, `Sir`; 32 a run of two or more, `The Rt Hon`) and a suffix in 320 (`MP`, `OBE`, `CBE`, `QC`). The population for titled British names. The Commission publishes its documents under the Open Government Licence v3.0, but the register states no reuse terms of its own, which the credit line says | the family-comma shape (2 names hold a comma) and case (9 are in capitals); small donors, since a donation is reported only above a threshold that depends on the donee; names outside the UK; names in any script but Latin | | `psc-individuals` | the people with significant control of UK companies, and the beneficial owners of overseas entities, from Companies House's daily snapshot (33 zipped parts of JSON lines, 2.44 GB on 2026-10-22): each person's display name, the four boxes the filer typed it into (title, forename, middle name, surname) and the stated nationality, nothing else. A record is a distinct filed name, kept one in eight by a hash of the record (1,040,708 of 8,322,939 on 2026-10-22), a size the machinery can hold in memory. A local corpus whose names carry their fields, scored by `local_agreement.py`, as `orcid-researchers` is | names nobody filed into boxes, so almost no comma (838 of 1,040,708 on 2026-10-22), no credential run and almost no suffix; a display name that is not its boxes joined, which the fetch drops (2 of 14,025,738 individuals on 2026-10-22) | | `orcid-researchers` | the public names of ORCID iD holders, researchers worldwide, from ORCID's Public Data File 2025 (`ORCID_2025_10_summaries.tar.gz`, 46.3 GB, data as of 2025-10-17), read as a stream and never stored: each line of its extract is the two boxes the holder typed their own name into (given names, family name), the credit name they publish under where public, other names, and a country, taken from their public address, else the first employment that names one, else the first education that does; a corpus record keeps the boxes, the credit name and the country. The fetch caches an extract of every summary with a public name (24,990,909 of 26,078,951 on 2026-10-27, 1,086,356 private and 1,686 empty), 369 MB, and the corpus keeps at most `ORCID_CAP` (20,000) records a country, the lowest by a hash of the record, so a change to the cap re-reads the extract rather than the 46 GB file: 1,831,533 records: 250 countries, 65 of them at the cap, and the records with no country (14,293,215 of the extract, 57%) as one more group of 20,000, counted in that total. A record's name is its boxes joined, given first; the credit name is a second corpus key (125,960 kept records hold one). Scored by `local_agreement.py` by country, the boxes as the holder split them. CC0, with ORCID asking for credit and respect for the privacy of the people in it | frequency, since the cap flattens every large country to 20,000; people outside research; name order, which ORCID does not record, so a holder who typed a family-first name into the given box reads as a disagreement; names with no family name and names whose family name is private, which look alike (1,308,200 of the extract hold no family name); titles, credentials and suffixes, which the form has no box for | +| `patentsview-inventors` | the inventor of every patent the USPTO granted from 1976 to 2025, from PatentsView's granted-patent tables on the USPTO Open Data Portal (release to 2025-12-31): each inventor's name in the filing's two boxes (first name, last name) and the country of the inventor's address. A record is a distinct name in a country, with the number of people who filed under it (`people`, counted by PatentsView's disambiguated inventor id, so a prolific inventor weighs once): 24,037,380 inventor rows, one per inventor per patent, held 5,319,206 such names on 2026-10-28, 99.3% of rows with a country. The extract keeps every name with its count, the corpus at most 20,000 names a country (657,991 records: 202 countries, 23 of them at the cap, and the records with no country as one more group of 20,000, counted in that total). Split by a professional filer, the applicant's attorney, rather than the bearer: `parse()` agreed with 94.7% of the records, 99.8% in Japan, 99.5% in China and 99.0% in the US, so it reads beside `orcid-researchers` as how a name is written against how its bearer types it. A comma stands in the last box of 92,503 of the extract's names, against 129 in the first, before suffixes and legal roles typed after a deceased inventor's name: the text after it is exactly `Jr.` in 55,459 names, `III` in 14,227, `deceased` in 4,635, `heir` in 514, `legal representative` in 393, `executrix` in 328 and `administrator` in 288 (2026-10-28). CC BY 4.0, credit PatentsView; the portal needs an API key | names in any script but Latin, since a filing is romanized; titles and credentials, which a filing has no place for; names outside patenting, which is mostly engineering and science, and its countries' shares are of patents filed in the US, not of their populations | | `jrc-names` | the person entities of the EU Joint Research Centre's JRC-Names, each with the spellings of its name harvested from news text in many languages and scripts: a record is one distinct spelling, with its entity and its script. The file (`entities.gzip`, 14.8 MB) is pinned by sha256 and dated by its Last-Modified, 2024-03-15. Of its 1,317,586 spellings kept, 1,203,634 are Latin, 66,673 Cyrillic, 17,074 Arabic (7,833 persons, 5,347 of them spelled in Latin too), 13,039 Katakana and 8,016 Han, and 218,822 persons have two or more spellings, 25,415 of them in two scripts or more (a spelling whose letters are in two scripts, or none, not counted as one). The population for spelling and transliteration variants of one name, read by the `groups` view, and for Arabic-script names. Licensed under the JRC-Names EULA, personal and non-transferable, with no redistribution or derivative works, so its views' output and its snapshot stay in the cache like every local corpus's, and here the licence, not only the names being real people's, is why | titles and credentials, since the names are harvested from running text (`parse()` read a title in 3,011 of its 1,315,051 names and a suffix in 1,940, 2026-10-24); the ASCII comma, which none holds (1,510 Arabic spellings hold an Arabic comma, U+060C, which the parser reads as one); the Indic scripts (Devanagari 827 spellings, Bengali 419, the others fewer); people the news did not name | | `nepal-candidates` | every candidate in Nepal's local elections of 2079 BS (2022), from the Election Commission's results site (result.election.gov.np), one file per local body: a record per distinct pair of the Commission's romanization (`name`, the one `parse()` reads) and the name in Devanagari as filed (`devanagari`). The corpus and the word index hold both, so a snapshot and a `name_usage.py` lookup read both scripts; `tally`, which counts what follows a comma, reads the romanized name. Of 126,981 pairs fetched 2026-10-24 (152,960 candidates in 753 files, 26 of whose romanizations were a placeholder such as `- - -` and are dropped), the romanized names carry surnames written as initials (`B.K.`, `K.C.`, `G.C.`), honorific words that are titles in front (`Guru`; `Shekh` until #752's phase 2j made it a title only in a title phrase) and `Md.`, Muhammad's abbreviation, which its period makes a title by shape; the Devanagari ones carry vowel signs and the virama, combining marks the letter-shape tests do not count as letters (decisions/nfd-shapes), and the joiners ZWJ and ZWNJ. The population for romanized South Asian names and for Devanagari. The site states no licence, which the credit line says | case and the comma, since the Commission's romanization is one consistent hand; names a person typed themselves; credentials and generational words; the parliament and provincial candidates, whose files carry Devanagari only; names outside Nepal | @@ -293,9 +294,11 @@ A live source has no fixed snapshot to pin, so the cached copy is the pin: `meta `orcid-researchers` streams the summaries file in HTTP Range requests through `gzip` and `tarfile`'s stream mode, and keeps nothing of it but the extract: a dropped connection reopens at the byte it reached, with `ORCID_RETRIES` attempts at each read (243 s of waits, about four minutes of outage before each attempt's own 60 s timeout, where the other sources' few attempts suit a small request), every part must be a 206 with the `Content-Range` asked for and the `Last-Modified` the first answer gave (a first answer stating none is refused), and once the tar ends the rest of the gzip is read so its CRC is checked, then the whole body's length and MD5 against the figshare listing's (`ORCID_SIZE`, `ORCID_MD5`). The extract replaces the cached one only after that check and after the fetch has read it back through every check a load makes, so a fetch refused at the end keeps the last good one; reading it back took 2 min 23 s on 2026-10-27. The first draft checked its shares only on load, after the rename, where a file past them would have replaced a good extract with one every load refused (found in review, 2026-10-27): a check a fetch's output must pass belongs before the rename, not only on the read. A fetch took 78 minutes on 2026-10-27, about 10 MB/s, with the XML parsed by a worker per CPU. Its summaries are parsed with `xml.etree` and a summary declaring a DTD or an entity is refused and counted, rather than adding a dependency for a safer parser; the fetch refuses past `ORCID_MAX_BAD` of summaries that do not parse or are refused that way, of tar members that are not summaries, or of countries that are not two capital letters (none of the four on 2026-10-27), and below the `ORCID_MIN_` shares of summaries giving a record, of records holding a family name and of records holding a country (95.8%, 94.8% and 42.8% on 2026-10-27), which catch a release whose layout moved under the reader once the pin moves. The count of summaries is pinned too (`ORCID_SUMMARIES`): `tarfile` ends a stream at a damaged header without raising, so a tar read only in part would pass its MD5. The extract is cut at `"\n"` only (`newline="\n"`), as `corpus_build.jsonl_lines` cuts, and streamed rather than read whole. `tarfile` keeps every member it has read in `TarFile.members`, even reading a stream, so the reader clears it after each one: before it did, the fetch held 2.3 GB a GiB into the file. An extract written by another version of the extraction is not read (`ORCID_EXTRACT_VERSION`, in its file name). Of the extract's records, the country came from the address in 2,842,439, an employment in 7,025,517 and an education in 829,738 (2026-10-27): a country is where a holder works more often than where they come from, so read a country's agreement as its institutions' population, not its nationals'. +`patentsview-inventors` is the one source whose fetch needs a credential: the USPTO Open Data Portal has needed a sign-in for every download since 2026-06-18, and its API answers a request without an API key with a 401 (2026-10-28); a key needs a USPTO.gov account verified through ID.me (data.uspto.gov, My ODP). The fetch reads it from `~/.config/nameparser-tools/uspto-odp-key` (`PV_KEY`), sends it as `X-API-KEY` to https uspto.gov hosts only, and drops it on a redirect anywhere else; a fetch without it is refused naming how to get one, and no test needs it. The portal lists each file's size and date but no checksum, so the release is pinned by those and by the sha256 of the bytes downloaded on 2026-10-28 (`PV_FILES`), and by the rows its inventor tables hold (`PV_ROWS`): every field of the TSVs is quoted, so a stray quote cannot fold rows together, and a count short of the pin is a table read in part. The raw inventor table gives the names as filed and the disambiguated one, row for row the same inventors, the inventor id and location; the reader takes them in step and refuses a pair of rows naming different inventor slots rather than join them. The fetch refuses past `PV_MAX_BAD` of rows not their header's width and below the `PV_MIN_` shares of rows naming a country and filling both boxes, and reads the extract back through every check a load makes before it replaces the cached one; the zips are deleted once it is built, and a run that fails keeps those already whole, so the next downloads only the rest. A download dropped mid-file is tried again from the start under the shared retry policy, and a key the portal refuses (401 or 403) is reported as the key's, naming how to replace it. A fetch took about five minutes on 2026-10-28, the three zips (1.75 GB) downloading in under one; the extract is 60 MB. The disambiguated table's attributed gender is read past. + `nepal-candidates` reads each local body's file through the site's own JSON handler, which asks for the site's double-submit check: the `CsrfToken` cookie a GET of the site sets, echoed in an `X-CSRF-Token` header, with the `Referer` its pages send. The site refuses a User-Agent holding `python` or `curl` with a 403 (2026-10-24), so this source sends a browser's form naming the tool (`NEPAL_USER_AGENT`) where the others send the repository's URL. A 403 mid-fetch is what a lapsed token gets, and renews the session once before it is raised; a 404 counts the file as missing, its id kept in `meta.json`'s detail, and with the empty files (a list of no candidates, as a site that blanked its files would answer) past `NEPAL_MAX_MISSING` of the `NEPAL_BODIES` bodies refuses the fetch, the empty ones' ids kept too. The election is over and its files final, so the candidates are pinned as the bodies are: a fetch reading every file must read exactly `NEPAL_CANDIDATES`, so a site that trimmed its files (to the winners, or to one post) is refused however well formed what is left; with files missing or empty the pin cannot be exact, and a shortfall of up to `NEPAL_MAX_SHORT` of it is accepted, so a few candidates trimmed elsewhere are not seen then. Every listed file answered on 2026-10-24: the 18 a survey on 2026-10-23 saw refused, among them ids 5551 to 5554, answered when asked again. A file is JSON, so a body cut short does not parse, and is retried as one cut in transit rather than read as a body with fewer candidates. The fetch refuses a lookup listing other than `NEPAL_BODIES` bodies (the election is over, so its files are final), a file that is not a list of candidates, too many candidates with no romanized or no Devanagari name (`NEPAL_MAX_NAMELESS`) or with a name in the wrong script (`NEPAL_MAX_WRONG_SCRIPT`: two columns swapped, or a romanization dropped), and too few distinct names in either column (`NEPAL_MIN_DISTINCT`). A record is a distinct pair, as `tse-donors`' are, not a candidate row: a body lists a candidate once per post, and a common name recurs across bodies. The `pairs` view refuses it, its records carrying no reference name: that view compares the letters of two spellings, and a romanization shares none with its Devanagari, so the Devanagari is kept under a key of its own. `meta.json` records the keys the corpus was built from (`corpus_keys`), and a corpus written under other keys is refused on load. A fetch took 28 minutes on 2026-10-24 at `NEPAL_DELAY` (0.3 s) a request. -`local_agreement.py` does the SCORE job for a local corpus. It parses each filed display name and compares the parser's roles with the boxes the source states (title, given, middle and family for `psc-individuals`; given and middle joined against the given-names box, then family, for `orcid-researchers`), folding case, periods, commas and spacing. A suffix the parser peels is compared inside the surname box: the display name joins the surname last, so a trailing suffix was typed into it whenever it is filled, with or without a comma (`Smith, Jr.`); the fold read the comma as a miss until 2026-10-28, which moved 83 `psc-individuals` records and 561 `orcid-researchers` records to agreeing and none away. Where a source has a title box, a title the parser reads otherwise, holding a word the vocabulary lists as neither a title nor a connective (a job typed as a title), is set aside as filer error rather than scored, and the report lists the commonest such titles as leads for the vocabulary. The report gives agreement overall and by the group the source states (nationality, country), misses per field, and the commonest disagreement shapes, as counts, word counts and the commonest set-aside titles; `--list N` prints moved records, names and all, for reading locally. Its record, `agreement.jsonl` beside the corpus, holds every record that did not wholly agree, so a re-run reports what moved. As with the local snapshot, check it from the branch against a base you name (`--base `, whose record is kept under `agreements/` beside the corpus, keyed by the fetch, the base's `nameparser/` files, the interpreter's minor version and the scorer, the ten most recent kept), or, alone, record the shared one from the base checkout (`--record --tree `) and run it from the branch; it refuses a record from another fetch, or one written by another version of the scorer, whose own change would otherwise read as the parser's moves. No CI run sees it. +`local_agreement.py` does the SCORE job for a local corpus. It parses each filed display name and compares the parser's roles with the boxes the source states (title, given, middle and family for `psc-individuals`; given and middle joined against the given-names box, then family, for `orcid-researchers`), folding case, periods, commas and spacing. A suffix the parser peels is compared inside the surname box: the display name joins the surname last, so a trailing suffix was typed into it whenever it is filled, with or without a comma (`Smith, Jr.`); the fold read the comma as a miss until 2026-10-28, which moved 83 `psc-individuals` records and 561 `orcid-researchers` records to agreeing and none away. Where a source has a title box, a title the parser reads otherwise, holding a word the vocabulary lists as neither a title nor a connective (a job typed as a title), is set aside as filer error rather than scored, and the report lists the commonest such titles as leads for the vocabulary. The report gives agreement overall and by the group the source states (nationality, country), misses per field, and the commonest disagreement shapes, as counts, word counts and the commonest set-aside titles; `--list N` prints moved records, names and all, for reading locally. Its record, `agreement.jsonl` beside the corpus, holds every record that did not wholly agree, so a re-run reports what moved. As with the local snapshot, check it from the branch against a base you name (`--base `, whose record is kept under `agreements/` beside the corpus, keyed by the fetch, the base's `nameparser/` files, the interpreter's minor version and the scorer, the ten most recent kept; the scorer's fingerprint reads only its own source's `FIELDED` entry, so adding a source leaves the others' records standing, which the whole table, hashed until `patentsview-inventors` was added on 2026-10-28, did not), or, alone, record the shared one from the base checkout (`--record --tree `) and run it from the branch; it refuses a record from another fetch, or one written by another version of the scorer, whose own change would otherwise read as the parser's moves. No CI run sees it. `pairs` reads a paired source (`tse-donors`, whose records carry the registry's name beside the filed one) as two spellings of one person, and sorts each pair by how the filed name departs from the reference: the first of its shapes that fits, from case, accents (case unchanged) and case and accents together, through words left out, shortened or misspelled, to few words in common. A word left out or added counts once per pair, and is listed only when ten pairs share it. `--agreement` also parses both spellings of every pair with a reference name that differs, and counts the pairs whose two readings end the family name in the same word (two readings with no family name share none). It lists the accents-only pairs read with two surnames by the two spellings' last words, the same ten-pair floor applied. @@ -331,7 +334,7 @@ Measured 2026-10-27 on a 12-CPU machine (load average 3 to 15 from other session uv run python tools/differential/measure.py --base ../master-checkout ``` -The steps: the committed snapshot (`snapshot.py`, the check `tests/v2/test_snapshot.py` makes), `cldr_agreement.py`, `snapshot.py --local --base` for every fetched local corpus, `local_agreement.py SOURCE --base` for every fetched source whose records carry their fields (`psc-individuals`, `orcid-researchers`), and `local_corpora.py alike --base` for every fetched source that pairs spellings. Only the local steps read `--base`: the first two compare the tree with the records committed on its branch, so on a branch that has re-recorded `snapshot.jsonl` the committed snapshot says no moves and the moves are in the PR's diff of that file, and the summary labels each step with what it compared against. A local corpus that is not fetched gets no step, the summary names it, and a run in which no step read `--base` exits 1. Each step's whole output goes to a file of its own under `--out`, by default a folder named for the time under the cache's `measure/`, outside the repository, since a local snapshot lists its moves by name. The summary, `summary.txt` beside them and printed, gives each step's status (no moves, moves, or failed with its exit code), its time, and its unindented lines (the first few and the last, where the verdicts are, up to `SUMMARY_LINES`, and any line saying part of a check was not measured), never the indented ones where the tools list names; a crash's traceback is indented below a line naming only its type, since its message may hold a name. A step that fails makes the run exit 1; one that finds moves does not. Each tool refuses with `SystemExit("message")`, which exits 1 as a move does, so measure runs every step through its own `--step` wrapper, which turns a message exit or an uncaught exception into exit 2: a refused base or a damaged corpus is reported as a failure, never as moves. +The steps: the committed snapshot (`snapshot.py`, the check `tests/v2/test_snapshot.py` makes), `cldr_agreement.py`, `snapshot.py --local --base` for every fetched local corpus, `local_agreement.py SOURCE --base` for every fetched source whose records carry their fields (`psc-individuals`, `orcid-researchers`, `patentsview-inventors`), and `local_corpora.py alike --base` for every fetched source that pairs spellings. Only the local steps read `--base`: the first two compare the tree with the records committed on its branch, so on a branch that has re-recorded `snapshot.jsonl` the committed snapshot says no moves and the moves are in the PR's diff of that file, and the summary labels each step with what it compared against. A local corpus that is not fetched gets no step, the summary names it, and a run in which no step read `--base` exits 1. Each step's whole output goes to a file of its own under `--out`, by default a folder named for the time under the cache's `measure/`, outside the repository, since a local snapshot lists its moves by name. The summary, `summary.txt` beside them and printed, gives each step's status (no moves, moves, or failed with its exit code), its time, and its unindented lines (the first few and the last, where the verdicts are, up to `SUMMARY_LINES`, and any line saying part of a check was not measured), never the indented ones where the tools list names; a crash's traceback is indented below a line naming only its type, since its message may hold a name. A step that fails makes the run exit 1; one that finds moves does not. Each tool refuses with `SystemExit("message")`, which exits 1 as a move does, so measure runs every step through its own `--step` wrapper, which turns a message exit or an uncaught exception into exit 2: a refused base or a damaged corpus is reported as a failure, never as moves. For a vocabulary word whose reading acts on a name position, `tools/vocab/word_report.py` gives the per-word view before and alongside this: the word in a grid of shapes, its incidence in every fetched local corpus, and its `name_usage.py` rows (`nameparser/vocab/README.md`, "Adding a word"). diff --git a/tools/differential/local_agreement.py b/tools/differential/local_agreement.py index 321a8475a..8c85fde77 100644 --- a/tools/differential/local_agreement.py +++ b/tools/differential/local_agreement.py @@ -122,6 +122,9 @@ def __post_init__(self) -> None: "surname": "family"}, group="nationality"), local_corpora.ORCID: Fielded( {"given": "given+middle", "family": "family"}, group="country", + group_cased=True), + local_corpora.PV: Fielded( + {"given": "given+middle", "family": "family"}, group="country", group_cased=True)} #: The fewest records sharing a set-aside title for the report to print #: it: a word typed into the title box by this many filers is no one @@ -300,7 +303,7 @@ def record_path(source: str) -> Path: def write_record(path: Path, local: local_corpora.Local, entries: dict[str, dict[str, Any]], tree: str) -> None: - header = {"raw_sha256": local.meta["sha256"], "scorer": scorer_fingerprint(), + header = {"raw_sha256": local.meta["sha256"], "scorer": scorer_fingerprint(local.source), "data_date": local.meta["data_date"], "tree": tree, "entries": len(entries)} lines = [corpus_build.jsonl_line(header)] @@ -339,7 +342,7 @@ def read_record(path: Path, local: local_corpora.Local) -> dict[str, dict[str, A if (count := recorded.get("entries", len(entries))) != len(entries): raise SystemExit(f"{path} holds {len(entries):,} entries where its header " f"counts {count!r}. {again}") - if recorded.get("scorer") != scorer_fingerprint(): + if recorded.get("scorer") != scorer_fingerprint(local.source): raise SystemExit(f"{path} was recorded by another version of this scorer, " f"so its moves would be the scorer's. {again}") return entries @@ -351,11 +354,12 @@ def read_record(path: Path, local: local_corpora.Local) -> dict[str, dict[str, A "key", "score", "_read_chunk") -def scorer_fingerprint() -> str: +def scorer_fingerprint(source: str) -> str: """local_corpora.code_fingerprint of the code that turns a parse into a record's entry and decides which records get one (_SCORER_CODE, a class by its methods), of corpus_build.jsonl_line, which writes an - entry's key, and of FIELDED.""" + entry's key, and of `source`'s own FIELDED entry, so a source added or + changed leaves the others' records standing.""" module = globals() functions: list[Callable[..., object]] = [] for name in _SCORER_CODE: @@ -364,7 +368,7 @@ class by its methods), of corpus_build.jsonl_line, which writes an if isinstance(target, type) else [target]) return local_corpora.code_fingerprint( [*functions, corpus_build.jsonl_line], - [repr(sorted((k, repr(v)) for k, v in FIELDED.items()))]) + [repr((source, FIELDED[source]))]) #: Where --base's kept records live, beside the shared one. @@ -378,7 +382,8 @@ def kept_path(source: str, local: local_corpora.Local, digest: str) -> Path: that matches is a record of this base.""" py = f"py{sys.version_info[0]}{sys.version_info[1]}" return (record_path(source).parent / KEPT - / f"{local.meta['sha256'][:12]}-{digest[:16]}-{py}-{scorer_fingerprint()[:12]}.jsonl") + / f"{local.meta['sha256'][:12]}-{digest[:16]}-{py}" + f"-{scorer_fingerprint(source)[:12]}.jsonl") def _score_base(root: Path, source: str, out: Path, jobs: int = 1) -> None: diff --git a/tools/differential/local_corpora.py b/tools/differential/local_corpora.py index 4069e0ab0..008b51102 100644 --- a/tools/differential/local_corpora.py +++ b/tools/differential/local_corpora.py @@ -39,7 +39,9 @@ with its entity and script; one distinct pair of a Nepali candidate's romanized name and the Devanagari one; one distinct ORCID record's two boxes joined, with the boxes, the credit name and the - country with where it came from. The counts are counted from here. + country with where it came from; one distinct PatentsView inventor + name in a country, its two boxes joined, with the boxes, the country + and how many people filed under it. The counts are counted from here. - corpus.jsonl -- the distinct names, in the corpus line format snapshot.py and compare.py --corpus read: each record's "name", and for nepal-candidates its Devanagari name too, and for @@ -70,8 +72,8 @@ (ctgov-officials), local_fec.py (fec-individuals), local_tse.py (tse-donors), local_tw.py (tw-donors), local_ukec.py (ukec-individuals), local_psc.py (psc-individuals), local_jrc.py (jrc-names), -local_nepal.py (nepal-candidates) and local_orcid.py -(orcid-researchers). +local_nepal.py (nepal-candidates), local_orcid.py +(orcid-researchers) and local_patentsview.py (patentsview-inventors). """ from __future__ import annotations @@ -112,6 +114,7 @@ from local_jrc import JRC, check_jrc, fetch_jrc, renew_jrc # noqa: E402 from local_nepal import check_nepal, fetch_nepal # noqa: E402 from local_orcid import ORCID, check_orcid, fetch_orcid, renew_orcid # noqa: E402 +from local_patentsview import PV, check_pv, fetch_pv, renew_pv # noqa: E402 from local_psc import PSC, check_psc, fetch_psc, renew_psc # noqa: E402 from local_tse import check_tse, fetch_tse # noqa: E402 from local_tw import fetch_tw # noqa: E402 @@ -228,6 +231,10 @@ def index(self) -> WordIndex: "of the people in it)", unit="records", fetch=fetch_orcid, renew=renew_orcid, check=check_orcid, corpus_keys=("name", "credit")), + PV: Source( + credit="PatentsView granted-patent inventors, 1976-2025, from the USPTO " + "Open Data Portal (CC BY 4.0; credit PatentsView)", + unit="names", fetch=fetch_pv, renew=renew_pv, check=check_pv), } diff --git a/tools/differential/local_patentsview.py b/tools/differential/local_patentsview.py new file mode 100644 index 000000000..80c94d1d6 --- /dev/null +++ b/tools/differential/local_patentsview.py @@ -0,0 +1,436 @@ +"""The patentsview-inventors local corpus: US patents' inventors, as their filings name them. + +patentsview-inventors -- the inventor of every patent the USPTO granted +from 1976 to 2025, from PatentsView's granted-patent tables on the +USPTO Open Data Portal: each inventor's name as the filing gives it, in +its two boxes (first name, last name), and the country of the +inventor's address. A local corpus whose names carry their fields +(local_agreement.FIELDED), so local_agreement.py scores the parser +against them. A filing is written by the applicant's attorney, so the +split is a professional filer's, not the inventor's own as in +orcid-researchers: the two read together tell how a name is written +from how its bearer types it. The name the parser reads is the two +boxes joined, first name first, as a filing writes it. + +A record is a distinct name in a country, with the number of people who +filed under it ("people", counted by PatentsView's disambiguated +inventor id): 24,037,380 inventor rows, one per inventor per patent, +held 5,319,206 such names of 4,294,034 inventors on 2026-10-28, +filed under 5,508,446 pairs of a name and an inventor. The +count is what lets a pack ask how common a word is in a country's names +(a frequency ORCID's capped corpus cannot give); it counts people, not +patents, so a prolific inventor weighs once. The extract keeps every +name; the corpus keeps at most PV_CAP a country, chosen by a hash of the +record as orcid-researchers chooses, for the tools that hold a corpus +in memory. + +The fetch downloads three of the product's files (PV_FILES) and reads +them without unpacking: the raw inventor table for the names as filed, +the disambiguated one, row for row the same inventors, for the inventor +id and location, and the disambiguated locations for the country. The +portal needs an API key, sent as X-API-KEY: a USPTO.gov account verified +through ID.me gets one (data.uspto.gov, My ODP), saved alone in PV_KEY, +read there and sent to https uspto.gov hosts only. The release is +pinned by each file's size, date and sha256 and the rows it holds; a +new quarter's is measured and the pin moved by hand. The zips are +deleted once the extract is built and read back, a failed run keeping +those already whole for the next; `fetch --refresh` downloads them +again. PatentsView licenses its data under CC BY 4.0 and +asks for credit; the inventors are living people, which keeps their +names in the cache. +""" +from __future__ import annotations + +import collections +import csv +import email.message +import gzip +import hashlib +import heapq +import io +import json +import os +import shutil +import sys +import urllib.error +import urllib.parse +import urllib.request +import zipfile +from collections import Counter +from collections.abc import Iterator +from pathlib import Path +from typing import IO, Any, cast + +import corpus_build +import local_core +from local_core import Fetched, _again, _backed_off, _pinned, _refuse_past, directory + + +PV = "patentsview-inventors" +PV_PRODUCT = "PVGPATDIS" +PV_API = "https://api.uspto.gov/api/v1/datasets/products" +#: The pinned release, as the portal listed it on 2026-10-28: grants to +#: this date, and for each file its size, its last modification as the +#: listing states it, and the sha256 of the bytes downloaded that day. +PV_RELEASE = "2025-12-31" +PV_RAW = "g_inventor_not_disambiguated.tsv.zip" +PV_INVENTORS = "g_inventor_disambiguated.tsv.zip" +PV_LOCATIONS = "g_location_disambiguated.tsv.zip" +PV_FILES = { + PV_RAW: (1_035_476_374, "2026-03-12 22:26:59", + "18bcef31301b216012a292242ea64393a5c88083ccb648972634df4492017c7b"), + PV_INVENTORS: (707_503_691, "2026-03-12 22:26:35", + "f1148d8c57e2c7d28062b4f572d9a9060eb537c4c62f54d69c63a047b7944385"), + PV_LOCATIONS: (2_697_159, "2026-03-12 22:27:08", + "526023908343f28491ebe6ea9d0568dfae219418f9c128d8a0c4f49e0ce9d6d9"), +} +#: The inventor rows the pinned release holds, each table the same: every +#: field is quoted, so a stray quote cannot fold rows together, and a +#: count short of this is a table read in part. +PV_ROWS = 24_037_380 +#: The columns read, by name: the tables state them in a header. +PV_COLUMNS = { + PV_RAW: ("patent_id", "inventor_sequence", "raw_inventor_name_first", + "raw_inventor_name_last"), + PV_INVENTORS: ("patent_id", "inventor_sequence", "inventor_id", "location_id"), + PV_LOCATIONS: ("location_id", "disambig_country"), +} +PV_KEY = Path.home() / ".config" / "nameparser-tools" / "uspto-odp-key" +#: The extract format, in the extract's file name. +PV_EXTRACT_VERSION = 1 +#: The most names a country keeps in the corpus. +PV_CAP = 20_000 +#: The share of rows a read skips as not the header's width before it +#: refuses the file (none of 24,037,380 on 2026-10-28), and the least +#: share whose location names a country (99.3%) and whose boxes are both +#: filled (all but 676). +PV_MAX_BAD = 1 / 10_000 +PV_MIN_COUNTRY = 0.9 +PV_MIN_BOTH = 0.9 +_CHUNK = 1 << 20 + + +def extract_path() -> Path: + """The extract cached for the pinned release.""" + return directory(PV) / f"inventors_{PV_RELEASE}.v{PV_EXTRACT_VERSION}.jsonl.gz" + + +def _get_a_key() -> str: + return ("sign in at data.uspto.gov with a USPTO.gov account verified through ID.me, " + f"copy the key under My ODP, and save it alone in {PV_KEY}, readable by you " + "only (chmod 600)") + + +def _key() -> str: + try: + key = PV_KEY.read_text(encoding="utf-8").strip() + except FileNotFoundError: + key = "" + if not key or any(c.isspace() for c in key): + raise ValueError(f"{PV} needs a USPTO Open Data Portal API key: {_get_a_key()}") + return key + + +def _uspto(url: str) -> bool: + parts = urllib.parse.urlsplit(url) + host = parts.hostname or "" + return parts.scheme == "https" and (host == "uspto.gov" or host.endswith(".uspto.gov")) + + +class _KeyStaysHome(urllib.request.HTTPRedirectHandler): + """A redirect off an https uspto.gov host goes without the key.""" + + def redirect_request(self, req: urllib.request.Request, fp: IO[bytes], code: int, msg: str, + headers: email.message.Message, newurl: str + ) -> urllib.request.Request | None: + new = super().redirect_request(req, fp, code, msg, headers, newurl) + if new is not None and not _uspto(newurl): + new.remove_header("X-api-key") + return new + + +def _open(url: str) -> IO[bytes]: + if not _uspto(url): + raise ValueError(f"{url} is not an https uspto.gov address: the key goes nowhere else") + request = urllib.request.Request(url, headers={ + "X-API-KEY": _key(), "User-Agent": local_core.USER_AGENT}) + try: + return cast(IO[bytes], + urllib.request.build_opener(_KeyStaysHome).open(request, timeout=120)) + except urllib.error.HTTPError as exc: + if exc.code not in (401, 403): + raise + exc.close() + raise ValueError(f"the portal refused the key in {PV_KEY} ({exc.code}): it may " + f"have expired or been revoked; for a new one, {_get_a_key()}" + ) from None + + +def _listed() -> dict[str, dict[str, Any]]: + """The product's files as the portal lists them, refused where a + pinned file is missing or not the size and date pinned: another + quarter's release, whose pin is moved by hand.""" + with _open(f"{PV_API}/{PV_PRODUCT}?includeFiles=true") as response: + data = corpus_build.parse_json(response.read(), PV_API) + found: dict[str, dict[str, Any]] = {} + + def walk(node: object) -> None: + if isinstance(node, dict): + if isinstance(node.get("fileName"), str): + found[node["fileName"]] = node + for value in node.values(): + walk(value) + elif isinstance(node, list): + for value in node: + walk(value) + walk(data) + for name, (size, modified, _) in PV_FILES.items(): + entry = found.get(name) + if entry is None or (entry.get("fileSize"), entry.get("fileLastModifiedDateTime")) != ( + size, modified): + held = entry and (entry.get("fileSize"), entry.get("fileLastModifiedDateTime")) + raise ValueError(f"{PV_PRODUCT} lists {name} as {held or 'missing'} where " + f"{(size, modified)} is pinned: another release, measured " + f"and pinned by hand in local_patentsview.PV_FILES") + return found + + +def _whole(path: Path, name: str) -> bool: + """Whether `path` is already the pinned file `name`, size and sha256.""" + size, _, sha = PV_FILES[name] + if not path.exists() or path.stat().st_size != size: + return False + digest = hashlib.sha256() + with path.open("rb") as held: + while chunk := held.read(_CHUNK): + digest.update(chunk) + return digest.hexdigest() == sha + + +def _download(entry: dict[str, Any], path: Path) -> None: + """One pinned file into `path` through a .part file, refused unless + it is the size and sha256 pinned; kept where a run before left it + whole, and tried again under local_core's retry policy where a + connection drops, each try from the start.""" + name = entry["fileName"] + if _whole(path, name): + return + size, _, sha = PV_FILES[name] + url = entry.get("fileDownloadURI") or f"{PV_API}/files/{PV_PRODUCT}/{name}" + partial = path.with_name(path.name + ".part") + for attempt in range(local_core.RETRIES): + print(f"downloading {name} ({size / 1e6:,.0f} MB) ...", file=sys.stderr) + digest, got = hashlib.sha256(), 0 + try: + with _open(url) as response, partial.open("wb") as out: + while chunk := response.read(_CHUNK): + out.write(chunk) + digest.update(chunk) + got += len(chunk) + except local_core._TRANSIENT as exc: + partial.unlink(missing_ok=True) + if not _backed_off(exc, attempt, url): + raise + continue + except BaseException: + partial.unlink(missing_ok=True) + raise + if (got, digest.hexdigest()) != (size, sha): + partial.unlink(missing_ok=True) + raise ValueError(f"{name} arrived as {got:,} bytes with sha256 " + f"{digest.hexdigest()}, where {size:,} and {sha} are pinned") + os.replace(partial, path) + return + raise AssertionError("unreachable: the last attempt raises") + + +def _rows(path: Path, name: str, counts: Counter[str]) -> Iterator[dict[str, str]]: + """A table's rows by the columns PV_COLUMNS names, read without + unpacking; a row not the header's width is counted and skipped.""" + with zipfile.ZipFile(path) as archive: + members = archive.namelist() + if len(members) != 1: + raise ValueError(f"{name} holds {len(members)} members where one is expected") + with archive.open(members[0]) as raw: + table = csv.reader(io.TextIOWrapper(raw, encoding="utf-8", newline=""), + delimiter="\t") + header = next(table) + missing = [c for c in PV_COLUMNS[name] if c not in header] + if missing: + raise ValueError(f"{name}'s header holds no {', '.join(missing)}") + at = [header.index(c) for c in PV_COLUMNS[name]] + for row in table: + if len(row) != len(header): + counts[f"{name}: rows not {len(header)} fields"] += 1 + continue + yield {c: row[i] for c, i in zip(PV_COLUMNS[name], at)} + + +def _extract(folder: Path, out: IO[str]) -> Counter[str]: + """Every distinct (first, last, country) of the raw table, with the + distinct inventors who filed under it, written to `out` as extract + lines; the counts. The raw and disambiguated tables are read in + step, row for row, and a pair of rows naming different inventor + slots is refused rather than joined.""" + counts: Counter[str] = Counter() + country_of = {r["location_id"]: r["disambig_country"] + for r in _rows(folder / PV_LOCATIONS, PV_LOCATIONS, counts)} + inventors = _rows(folder / PV_INVENTORS, PV_INVENTORS, counts) + people: dict[tuple[str, str, str], int] = {} + seen: set[int] = set() + for raw in _rows(folder / PV_RAW, PV_RAW, counts): + row = next(inventors, None) + slot = (raw["patent_id"], raw["inventor_sequence"]) + if row is None or (row["patent_id"], row["inventor_sequence"]) != slot: + raise ValueError(f"{PV_RAW} and {PV_INVENTORS} part at row {counts['rows'] + 1:,}:" + f" the two tables are not row for row the same inventors") + counts["rows"] += 1 + first = " ".join(raw["raw_inventor_name_first"].split()) + last = " ".join(raw["raw_inventor_name_last"].split()) + country = country_of.get(row["location_id"], "").strip().upper() + counts["rows_with_a_country"] += bool(country) + if not first or not last: + counts["rows_without_both_boxes"] += 1 + if not first and not last: + continue + key = (first, last, country) + # One inventor counts once under a name, however many patents. + filed = hash((key, row["inventor_id"])) + if filed not in seen: + seen.add(filed) + people[key] = people.get(key, 0) + 1 + if next(inventors, None) is not None: + raise ValueError(f"{PV_INVENTORS} holds more rows than {PV_RAW}") + for (first, last, country), n in people.items(): + out.write(corpus_build.jsonl_line( + {k: v for k, v in (("given", first), ("family", last), ("country", country)) if v} + | {"people": n}, sort_keys=True) + "\n") + counts["names"] = len(people) + counts["people_by_name"] = sum(people.values()) + return counts + + +def renew_pv() -> None: + """Download the pinned files (_download), build the extract + (_extract) and read it back through every check a load makes + (_pv_read) before it replaces the cached one. The zips are deleted + once it is in place; a run that fails keeps those already whole, so + the next downloads only the rest. The extract opens with a header of + the release, the files' sha256 and the counts.""" + target = extract_path() + folder = target.parent / "downloads" + body, staged = (target.with_name(target.name + end) for end in (".part", ".new")) + folder.mkdir(parents=True, exist_ok=True) + try: + listed = _listed() + for name in PV_FILES: + _download(listed[name], folder / name) + with gzip.open(body, "wt", encoding="utf-8", newline="\n") as out: + counts = _extract(folder, out) + header = corpus_build.jsonl_line({ + "release": PV_RELEASE, "files": {n: sha for n, (_, _, sha) in PV_FILES.items()}, + "counts": dict(counts)}) + # The header is a gzip member of its own ahead of the names', as + # orcid-researchers' is: the counts are known only at the end. + with staged.open("wb") as joined, body.open("rb") as names: + joined.write(gzip.compress(f"{header}\n".encode())) + shutil.copyfileobj(names, joined, _CHUNK) + _pv_read(staged) + os.replace(staged, target) + shutil.rmtree(folder, ignore_errors=True) + finally: + body.unlink(missing_ok=True) + staged.unlink(missing_ok=True) + + +def pv_record(line: dict[str, Any]) -> dict[str, str]: + """An extract line as a corpus record: the boxes joined first name + first as "name", the boxes, the country and the people, written as + text since a corpus record holds only text (local_corpora.read_records); + the extract keeps the number.""" + record = {k: line[k] for k in ("given", "family", "country") if line.get(k)} + record["name"] = " ".join(v for v in (line.get("given"), line.get("family")) if v) + record["people"] = str(line["people"]) + return record + + +def pv_kept(records: Iterator[dict[str, str]], cap: int) -> list[dict[str, str]]: + """At most `cap` records a country (none is a country of its own): + those whose line hashes lowest, so a country under the cap is kept + whole and the same names are kept whatever else the file holds.""" + heaps: dict[str, list[tuple[int, str]]] = collections.defaultdict(list) + for record in records: + line = corpus_build.jsonl_line(record, sort_keys=True) + rank = -int(hashlib.sha256(line.encode()).hexdigest()[:16], 16) + heap = heaps[record.get("country", "")] + if len(heap) < cap: + heapq.heappush(heap, (rank, line)) + elif rank > heap[0][0]: + heapq.heapreplace(heap, (rank, line)) + return [json.loads(line) for heap in heaps.values() for _, line in sorted(heap)] + + +def _pv_read(path: Path) -> Fetched: + """The corpus from the cached extract (pv_record, pv_kept), or + ValueError naming the refresh where it is not an extract of the + pinned release, where it holds another number of names than its + header counts, where its rows are not PV_ROWS, or past PV_MAX_BAD + of them were malformed, or fewer than PV_MIN_COUNTRY named a country + or PV_MIN_BOTH filled both boxes. The extract is cut at "\\n" only, + as corpus_build.jsonl_lines cuts, and streamed rather than read + whole.""" + lines_read = 0 + + def read(lines: Iterator[str]) -> Iterator[dict[str, str]]: + nonlocal lines_read + for line in lines: + lines_read += 1 + yield pv_record(json.loads(line)) + + try: + with gzip.open(path, "rt", encoding="utf-8", newline="\n") as lines: + head = json.loads(next(lines)) + counts = head["counts"] + if (head.get("release") != PV_RELEASE + or head.get("files") != {n: sha for n, (_, _, sha) in PV_FILES.items()} + or not isinstance(counts, dict) + or not all(type(n) is int for n in counts.values())): + raise ValueError("its header does not describe the pinned release") + records = pv_kept(read(lines), PV_CAP) + except (OSError, EOFError, StopIteration, ValueError, KeyError, TypeError) as exc: + raise ValueError(f"{path} is not an extract this tool wrote ({exc}){_again(PV)}" + ) from None + rows = counts.get("rows", 0) + if rows != PV_ROWS or lines_read != counts.get("names", -1): + raise ValueError(f"{path} holds {lines_read:,} names of {rows:,} rows, where its " + f"header counts {counts.get('names', 0):,} names and {PV_ROWS:,} " + f"rows are pinned: a table was read in part{_again(PV)}") + why = "the files are not the PatentsView tables this tool reads" + _refuse_past([(n, rows, PV_MAX_BAD, what) for what, n in counts.items() + if what.endswith("fields")], why) + for count, least, what in ((counts.get("rows_with_a_country", 0), PV_MIN_COUNTRY, + "rows name a country"), + (rows - counts.get("rows_without_both_boxes", 0), PV_MIN_BOTH, + "rows fill both boxes")): + if count < rows * least: + raise ValueError(f"only {count:,} of {rows:,} {what}: {why}") + by_country = Counter(r["country"] for r in records if "country" in r) + return Fetched(records, PV_RELEASE, { + "kept": _kept(), **{key: counts[key] for key in sorted(counts)}, + "countries": len(by_country), + "countries_at_the_cap": sum(n == PV_CAP for n in by_country.values())}) + + +def _kept() -> str: + return f"release {PV_RELEASE}, at most {PV_CAP:,} names a country" + + +def fetch_pv() -> Fetched: + """The corpus (_pv_read), read from the extract cached beside it, + downloaded and built when missing and again by `fetch --refresh`.""" + if not extract_path().exists(): + renew_pv() + return _pv_read(extract_path()) + + +check_pv = _pinned("kept", _kept, "keeps {held}, and this tool keeps {wanted}")