Skip to content

Add tools/name_usage.py: count a word's bearers as given and family name - #651

Merged
derek73 merged 5 commits into
masterfrom
feat/name-usage-tool
Oct 12, 2026
Merged

derek73 merged 5 commits into
masterfrom
feat/name-usage-tool

Conversation

@derek73

@derek73 derek73 commented Oct 12, 2026 •

Copy link
Copy Markdown
Owner

Adds tools/name_usage.py, a dev-only lookup that answers "how often is this word borne as a given name, and as a family name?" with counts instead of memory.

Why

Vocabulary claims are weighed by decisions.md#vocabulary-collisions: a word the parser claims by position (titles, never-given particles) is safe to claim outright only if it is not borne as a name in the position the claim acts on (C-i; #salutation-titles records the one TITLES-only exception, decided on counts). That check has often been run from memory, which misses what the data shows at once. shehu, a title, is also an Albanian and Nigerian surname: 561 bearers in the 2010 US Census, and 180 people on Wikidata, mostly Nigerian and Albanian citizens.

What it does

uv run python tools/name_usage.py shehu greve marshal
uv run python tools/name_usage.py McDonald     # pass names as written
uv run python tools/name_usage.py --no-wikidata knight   # offline, downloads nothing
uv run python tools/name_usage.py --json justice

One row per source, given and family counts, plus Wikidata's top countries of citizenship per role (which tells you whether a surname collision is in a culture that writes the surname first):

source license a count means how it's fetched
US Census 2010 surnames public domain bearers; unlisted shows <100 downloaded once into a cache outside the repo, validated before caching
US SSA given names, 1880 onward CC0 births summed over years, a floor (+); unlisted shows <5/yr by hand: ssa.gov returns 403 to scripts, so the tool says where to save the zip
Wikidata P735 / P734 CC0 people with the statement, a lower bound (+); a zero proves nothing live SPARQL, every spelling in one query

It reports; it does not decide. Every row carries a status (ok, unrepresentable, not downloaded, failed), so "no answer" is never shown as 0. The exit status is 2 when any source failed. The cache lives in $NAMEPARSER_TOOLS_CACHE or ~/.cache/nameparser-tools, so no third-party data enters the tree.

Notes

  • Both US lists spell names in ASCII letters only (O'Brien → OBRIEN). The fold transliterates letters NFKD leaves whole (Sørensen → SORENSEN, Strauß → STRAUSS) and drops apostrophes, including modifier letters like the ʻokina. A letter with no ASCII spelling makes the word unrepresentable, never a partial lookup of a different name.
  • Wikidata labels match exactly, in four spellings: as written, lower-case, capitalized per word, and title case. No rule rebuilds an internal capital, so mcdonald finds 0 where McDonald finds 1,968. The report lists the spellings tried, and the docstring and AGENTS.md say to pass names as written.
  • A country with no English label is named by its Q-id, and an "unknown value" citizenship is named as unknown. Only a person with no citizenship statement at all counts as "no citizenship stated".
  • I tried a label-position search ("does the word open people's names?"), which would answer the rule's position question directly. It timed out on the query service, so it's not included.
  • AGENTS.md's "Adding a word to a config set" and "Titles permanently shadow first names" paragraphs now point at the tool.

Review

Ran /pr-review-toolkit:review-pr (code, tests, comments, errors, types), fixed what it found in 088d44d, then reviewed that fix commit, which turned up four defects the fix had introduced; those are fixed in bd76515. A simplification pass followed (8cd6781). The findings that mattered were all false zeros:

  • The original fold deleted ø/ß/Ł, so Sørensen reported 0 against a Census count of 28,069.
  • A wrong SSA zip read as empty, so every word reported 0.
  • Summing per-spelling queries double-counted people.
  • The first fix's SSA validation sat outside any try, so it crashed the run.

Tests

tests/v2/test_name_usage_tool.py: 33 offline tests behind an autouse guard that fails any network call (with a test proving the guard bites). Negative controls re-taken after the last test commit:

  • no transliteration: 6 tests fail
  • lossy fold allowed: 1 fails
  • SSA year check off: 2 fail
  • role check off: 1 fails
  • modifier letters not skipped: 1 fails
  • SSA load unguarded: 1 fails

Full suite at the first commit: 15,008 passed. Ruff and mypy are clean, and the tool also passes mypy --strict.

🤖 Generated with Claude Code

derek73 and others added 2 commits October 11, 2026 20:17
…r family name

Vocabulary membership turns on decisions.md#vocabulary-collisions: a
word joins a positionally-claimed set only if it is not borne as a name
in the position the claim acts on. That check was run from memory, and
memory missed what the data shows at once -- shehu is an Albanian
surname as well as a Hausa given name, and herra (a title since #606)
is a US Census surname with about 640 bearers.

The tool reports counts per source and role: US Census 2010 surnames
(downloaded once into a cache outside the repo), SSA given names (a
hand download, since ssa.gov refuses scripts), and live Wikidata given-
and family-name statements with countries of citizenship, a lower bound.
It reports; it does not decide.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@derek73 derek73 self-assigned this Oct 12, 2026
@derek73 derek73 added this to the 2.5 milestone Oct 12, 2026
@codecov

codecov Bot commented Oct 12, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.02%. Comparing base (73a6b4a) to head (8cd6781).

Additional details and impacted files
@@           Coverage Diff           @@
##           master     #651   +/-   ##
=======================================
  Coverage   99.02%   99.02%           
=======================================
  Files          46       46           
  Lines        4513     4513           
=======================================
  Hits         4469     4469           
  Misses         44       44           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

derek73 and others added 3 commits October 11, 2026 20:30
Five reviewers converged on one failure class -- a count that reads as
'nobody bears this name' when the tool simply failed to look:

- The ASCII fold deleted letters NFKD leaves whole (ø, ß, Ł, Æ, Đ), so
  Sørensen was looked up as SRENSEN and reported 0 against a Census
  count of 28,069 (and Æsir looked up SIR). Transliterate them first,
  and treat any letter still without an ASCII spelling as
  unrepresentable rather than dropping it.
- SSA reported 0 for a word it cannot spell (关); it now says
  unrepresentable, as the Census path already did.
- A wrong or partial SSA zip (the state-level file, a re-zipped folder)
  read as empty, so every word reported 0. Members now match by
  basename in any case, and fewer than 100 year files is refused.
- Case variants were queried separately and summed, counting twice a
  person reached through two spellings (de: 184 against 182). All
  spellings now go in one VALUES block; _merge is gone. Lowercase
  input cannot rebuild an internal capital (mcdonald finds 0 where
  McDonald finds 1,968), so the report lists the spellings tried and
  AGENTS.md says to pass names as written.
- Every source's 0 meant 'below the listing threshold', not zero:
  Usage now carries a status, a floor (<100, <5/yr) and a lower_bound
  flag, rendered and in --json.
- A country with no English label counted its people as stateless; it
  is now named by its Q-id.
- A malformed Wikidata response crashed the run and lost every word;
  an unusable download stayed cached forever; --no-wikidata still
  downloaded. Each now reports, and the exit status is 2 when a source
  failed.

The docstring also claimed herra was a collision memory had missed; it
was shipped knowingly under decisions.md#salutation-titles with
recorded counts. Corrected, with shehu as the example. Tests grow from
12 to 31, behind an autouse guard that fails any network call.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The new SSA validation raised outside any try, so a state-level zip
  saved in place of the national file crashed the run, losing the
  Census answer and exiting 1. Both cached sources now load under the
  same handling: failed per word, the run continues, exit 2.
- The strict fold refused modifier letters (category Lm, e.g. the
  okina in Kaʻahumanu) as letters with no ASCII spelling, a regression
  from the original fold. They are dropped like an apostrophe.
- An 'unknown value' citizenship (a blank-node IRI) fell into the
  no-citizenship bucket; it is now named as unknown.
- A truncated Census download (http.client.IncompleteRead, not an
  OSError) crashed the run; it is caught with the rest.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
One loader helper in place of two identical try blocks in main(); the
role check in read_wikidata runs once; a test reads its rows by name.
No behavior change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@derek73
derek73 merged commit 774b19f into master Oct 12, 2026
12 checks passed
@derek73
derek73 deleted the feat/name-usage-tool branch October 13, 2026 06:59
@derek73 derek73 modified the milestones: 2.5, 2.4.1 Oct 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant