Repository navigation
Add tools/name_usage.py: count a word's bearers as given and family name - #651
Merged
Merged
Conversation
…r family name Vocabulary membership turns on decisions.md#vocabulary-collisions: a word joins a positionally-claimed set only if it is not borne as a name in the position the claim acts on. That check was run from memory, and memory missed what the data shows at once -- shehu is an Albanian surname as well as a Hausa given name, and herra (a title since #606) is a US Census surname with about 640 bearers. The tool reports counts per source and role: US Census 2010 surnames (downloaded once into a cache outside the repo), SSA given names (a hand download, since ssa.gov refuses scripts), and live Wikidata given- and family-name statements with countries of citizenship, a lower bound. It reports; it does not decide. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #651 +/- ##
=======================================
Coverage 99.02% 99.02%
=======================================
Files 46 46
Lines 4513 4513
=======================================
Hits 4469 4469
Misses 44 44 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Five reviewers converged on one failure class -- a count that reads as 'nobody bears this name' when the tool simply failed to look: - The ASCII fold deleted letters NFKD leaves whole (ø, ß, Ł, Æ, Đ), so Sørensen was looked up as SRENSEN and reported 0 against a Census count of 28,069 (and Æsir looked up SIR). Transliterate them first, and treat any letter still without an ASCII spelling as unrepresentable rather than dropping it. - SSA reported 0 for a word it cannot spell (关); it now says unrepresentable, as the Census path already did. - A wrong or partial SSA zip (the state-level file, a re-zipped folder) read as empty, so every word reported 0. Members now match by basename in any case, and fewer than 100 year files is refused. - Case variants were queried separately and summed, counting twice a person reached through two spellings (de: 184 against 182). All spellings now go in one VALUES block; _merge is gone. Lowercase input cannot rebuild an internal capital (mcdonald finds 0 where McDonald finds 1,968), so the report lists the spellings tried and AGENTS.md says to pass names as written. - Every source's 0 meant 'below the listing threshold', not zero: Usage now carries a status, a floor (<100, <5/yr) and a lower_bound flag, rendered and in --json. - A country with no English label counted its people as stateless; it is now named by its Q-id. - A malformed Wikidata response crashed the run and lost every word; an unusable download stayed cached forever; --no-wikidata still downloaded. Each now reports, and the exit status is 2 when a source failed. The docstring also claimed herra was a collision memory had missed; it was shipped knowingly under decisions.md#salutation-titles with recorded counts. Corrected, with shehu as the example. Tests grow from 12 to 31, behind an autouse guard that fails any network call. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The new SSA validation raised outside any try, so a state-level zip saved in place of the national file crashed the run, losing the Census answer and exiting 1. Both cached sources now load under the same handling: failed per word, the run continues, exit 2. - The strict fold refused modifier letters (category Lm, e.g. the okina in Kaʻahumanu) as letters with no ASCII spelling, a regression from the original fold. They are dropped like an apostrophe. - An 'unknown value' citizenship (a blank-node IRI) fell into the no-citizenship bucket; it is now named as unknown. - A truncated Census download (http.client.IncompleteRead, not an OSError) crashed the run; it is caught with the rest. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
One loader helper in place of two identical try blocks in main(); the role check in read_wikidata runs once; a test reads its rows by name. No behavior change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
tools/name_usage.py, a dev-only lookup that answers "how often is this word borne as a given name, and as a family name?" with counts instead of memory.Why
Vocabulary claims are weighed by decisions.md#vocabulary-collisions: a word the parser claims by position (titles, never-given particles) is safe to claim outright only if it is not borne as a name in the position the claim acts on (C-i; #salutation-titles records the one TITLES-only exception, decided on counts). That check has often been run from memory, which misses what the data shows at once.
shehu, a title, is also an Albanian and Nigerian surname: 561 bearers in the 2010 US Census, and 180 people on Wikidata, mostly Nigerian and Albanian citizens.What it does
One row per source, given and family counts, plus Wikidata's top countries of citizenship per role (which tells you whether a surname collision is in a culture that writes the surname first):
<100+); unlisted shows<5/yr+); a zero proves nothingIt reports; it does not decide. Every row carries a status (
ok,unrepresentable,not downloaded,failed), so "no answer" is never shown as 0. The exit status is 2 when any source failed. The cache lives in$NAMEPARSER_TOOLS_CACHEor~/.cache/nameparser-tools, so no third-party data enters the tree.Notes
O'Brien→OBRIEN). The fold transliterates letters NFKD leaves whole (Sørensen→SORENSEN,Strauß→STRAUSS) and drops apostrophes, including modifier letters like the ʻokina. A letter with no ASCII spelling makes the word unrepresentable, never a partial lookup of a different name.mcdonaldfinds 0 whereMcDonaldfinds 1,968. The report lists the spellings tried, and the docstring and AGENTS.md say to pass names as written.Review
Ran
/pr-review-toolkit:review-pr(code, tests, comments, errors, types), fixed what it found in 088d44d, then reviewed that fix commit, which turned up four defects the fix had introduced; those are fixed in bd76515. A simplification pass followed (8cd6781). The findings that mattered were all false zeros:ø/ß/Ł, soSørensenreported 0 against a Census count of 28,069.Tests
tests/v2/test_name_usage_tool.py: 33 offline tests behind an autouse guard that fails any network call (with a test proving the guard bites). Negative controls re-taken after the last test commit:Full suite at the first commit: 15,008 passed. Ruff and mypy are clean, and the tool also passes
mypy --strict.🤖 Generated with Claude Code