You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit e02cbdd
Browse filesBrowse the repository at this point in the historyBrowse files
Read a comma in a box as a space when scoring local agreement
local_agreement.fold dropped periods but kept commas, so a suffix typed
into the surname box after a comma (`Smith, Jr.`) scored as a miss
though the parser reads the name right (family Smith, suffix Jr.).
PatentsView's US inventors showed it as their commonest disagreement.
A fold applied to both sides can only turn misses into agreements:
over every record holding a comma, 83 psc-individuals records and 561
orcid-researchers records now agree, none move away, and 137 more have
fewer fields wrong. The scorer's fingerprint moves, so both shared
records are re-recorded.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copy file name to clipboardExpand all lines: tools/differential/README.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -295,7 +295,7 @@ A live source has no fixed snapshot to pin, so the cached copy is the pin: `meta
295
295
296
296
`nepal-candidates` reads each local body's file through the site's own JSON handler, which asks for the site's double-submit check: the `CsrfToken` cookie a GET of the site sets, echoed in an `X-CSRF-Token` header, with the `Referer` its pages send. The site refuses a User-Agent holding `python` or `curl` with a 403 (2026-10-24), so this source sends a browser's form naming the tool (`NEPAL_USER_AGENT`) where the others send the repository's URL. A 403 mid-fetch is what a lapsed token gets, and renews the session once before it is raised; a 404 counts the file as missing, its id kept in `meta.json`'s detail, and with the empty files (a list of no candidates, as a site that blanked its files would answer) past `NEPAL_MAX_MISSING` of the `NEPAL_BODIES` bodies refuses the fetch, the empty ones' ids kept too. The election is over and its files final, so the candidates are pinned as the bodies are: a fetch reading every file must read exactly `NEPAL_CANDIDATES`, so a site that trimmed its files (to the winners, or to one post) is refused however well formed what is left; with files missing or empty the pin cannot be exact, and a shortfall of up to `NEPAL_MAX_SHORT` of it is accepted, so a few candidates trimmed elsewhere are not seen then. Every listed file answered on 2026-10-24: the 18 a survey on 2026-10-23 saw refused, among them ids 5551 to 5554, answered when asked again. A file is JSON, so a body cut short does not parse, and is retried as one cut in transit rather than read as a body with fewer candidates. The fetch refuses a lookup listing other than `NEPAL_BODIES` bodies (the election is over, so its files are final), a file that is not a list of candidates, too many candidates with no romanized or no Devanagari name (`NEPAL_MAX_NAMELESS`) or with a name in the wrong script (`NEPAL_MAX_WRONG_SCRIPT`: two columns swapped, or a romanization dropped), and too few distinct names in either column (`NEPAL_MIN_DISTINCT`). A record is a distinct pair, as `tse-donors`' are, not a candidate row: a body lists a candidate once per post, and a common name recurs across bodies. The `pairs` view refuses it, its records carrying no reference name: that view compares the letters of two spellings, and a romanization shares none with its Devanagari, so the Devanagari is kept under a key of its own. `meta.json` records the keys the corpus was built from (`corpus_keys`), and a corpus written under other keys is refused on load. A fetch took 28 minutes on 2026-10-24 at `NEPAL_DELAY` (0.3 s) a request.
297
297
298
-
`local_agreement.py` does the SCORE job for a local corpus. It parses each filed display name and compares the parser's roles with the boxes the source states (title, given, middle and family for `psc-individuals`; given and middle joined against the given-names box, then family, for `orcid-researchers`), folding case, periods and spacing. A suffix the parser peels is compared inside the surname box: the display name joins the surname last, so a trailing suffix was typed into it whenever it is filled. Where a source has a title box, a title the parser reads otherwise, holding a word the vocabulary lists as neither a title nor a connective (a job typed as a title), is set aside as filer error rather than scored, and the report lists the commonest such titles as leads for the vocabulary. The report gives agreement overall and by the group the source states (nationality, country), misses per field, and the commonest disagreement shapes, as counts, word counts and the commonest set-aside titles; `--list N` prints moved records, names and all, for reading locally. Its record, `agreement.jsonl` beside the corpus, holds every record that did not wholly agree, so a re-run reports what moved. As with the local snapshot, check it from the branch against a base you name (`--base <checkout>`, whose record is kept under `agreements/` beside the corpus, keyed by the fetch, the base's `nameparser/` files, the interpreter's minor version and the scorer, the ten most recent kept), or, alone, record the shared one from the base checkout (`--record --tree <checkout>`) and run it from the branch; it refuses a record from another fetch, or one written by another version of the scorer, whose own change would otherwise read as the parser's moves. No CI run sees it.
298
+
`local_agreement.py` does the SCORE job for a local corpus. It parses each filed display name and compares the parser's roles with the boxes the source states (title, given, middle and family for `psc-individuals`; given and middle joined against the given-names box, then family, for `orcid-researchers`), folding case, periods, commas and spacing. A suffix the parser peels is compared inside the surname box: the display name joins the surname last, so a trailing suffix was typed into it whenever it is filled, with or without a comma (`Smith, Jr.`); the fold read the comma as a miss until 2026-10-28, which moved 83 `psc-individuals` records and 561 `orcid-researchers` records to agreeing and none away. Where a source has a title box, a title the parser reads otherwise, holding a word the vocabulary lists as neither a title nor a connective (a job typed as a title), is set aside as filer error rather than scored, and the report lists the commonest such titles as leads for the vocabulary. The report gives agreement overall and by the group the source states (nationality, country), misses per field, and the commonest disagreement shapes, as counts, word counts and the commonest set-aside titles; `--list N` prints moved records, names and all, for reading locally. Its record, `agreement.jsonl` beside the corpus, holds every record that did not wholly agree, so a re-run reports what moved. As with the local snapshot, check it from the branch against a base you name (`--base <checkout>`, whose record is kept under `agreements/` beside the corpus, keyed by the fetch, the base's `nameparser/` files, the interpreter's minor version and the scorer, the ten most recent kept), or, alone, record the shared one from the base checkout (`--record --tree <checkout>`) and run it from the branch; it refuses a record from another fetch, or one written by another version of the scorer, whose own change would otherwise read as the parser's moves. No CI run sees it.
299
299
300
300
`pairs` reads a paired source (`tse-donors`, whose records carry the registry's name beside the filed one) as two spellings of one person, and sorts each pair by how the filed name departs from the reference: the first of its shapes that fits, from case, accents (case unchanged) and case and accents together, through words left out, shortened or misspelled, to few words in common. A word left out or added counts once per pair, and is listed only when ten pairs share it. `--agreement` also parses both spellings of every pair with a reference name that differs, and counts the pairs whose two readings end the family name in the same word (two readings with no family name share none). It lists the accents-only pairs read with two surnames by the two spellings' last words, the same ten-pair floor applied.
0 commit comments