From 1a977ccc389c1617934a006458d920e159819abf Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Thu, 5 Nov 2026 13:51:14 -0800 Subject: [PATCH 1/4] Set a joined credential apart in brackets, as its split spelling is (#1088) rules/S1's set-apart word now asks `suffix_as_written`, the question a bare word is asked, so `John Smith (MD/PhD)`, `John (MD-PhD) Smith` and `Smith (MD/PhD), John` read suffix where they read nickname, as `(MD, PhD)` already did. A member of the ambiguous class written with a period inside it is asked first and answered "member", so rules/S13 keeps its position test; post_rules' two copies of that dotted test go, and the report on a member left the nickname leaves off a comma the clause keeps (`Jean (M.A,) Smith` reports suffix-or-nickname). Snapshot 0 of 6,939 moves; grid 447 of 1,260; local corpora: ctgov 3, FEC 1, PSC and ORCID none (decisions/S1, 2026-11-05). Co-Authored-By: Claude Opus 5.5 --- ...-bracketed-credential-is-not-a-nickname.md | 4 ++ ...bracketed-member-settled-by-its-writing.md | 1 + ...-bracketed-credential-is-not-a-nickname.md | 7 ++- ...bracketed-member-settled-by-its-writing.md | 3 +- nameparser/_pipeline/_extract.py | 3 +- nameparser/_pipeline/_post_rules.py | 29 +++++----- nameparser/_pipeline/_vocab.py | 29 ++++++---- tests/v2/cases.py | 53 +++++++++++++++++ tests/v2/pipeline/test_post_rules.py | 57 ++++++++++++++++++- tests/v2/rules_budget.json | 4 +- tests/v2/test_cases.py | 8 ++- tests/v2/test_layering.py | 7 ++- 12 files changed, 168 insertions(+), 37 deletions(-) diff --git a/docs/design/decisions/S1-bracketed-credential-is-not-a-nickname.md b/docs/design/decisions/S1-bracketed-credential-is-not-a-nickname.md index 6351f35f0..17c5c20ee 100644 --- a/docs/design/decisions/S1-bracketed-credential-is-not-a-nickname.md +++ b/docs/design/decisions/S1-bracketed-credential-is-not-a-nickname.md @@ -3,6 +3,7 @@ ## Standing - A clause wholly of words the vocabulary reads as no name is set apart from the name: it is no nickname and no maiden name, each word takes the role its vocabulary gives it (a title the title, a credential or generation the suffix, a word that is both the title in front of the name's first word and the suffix elsewhere), the words around it read as they would beside a nickname (a lone name word beside it the family name, N3), and a member of the ambiguous class is one of its words where its writing settles it after the name (S13): `John Smith (PhD, MBA)` and `John Smith (Ph.D., MBA)` → suffix, `John Smith (Dr)` → title, `John Smith (PhD) Withers` → family 'Withers', suffix 'PhD' — 2026-11-04 (Derek), #792. +- A credential or generation S3 reads joined by a slash or a hyphen is a word of such a clause, asked as the bare word is asked, a member of the ambiguous class first left to S13: `John Smith (MD/PhD)`, `John (MD-PhD) Smith` → suffix, `John Smith (MD/Bob)` and `(MA/PhD)` → nickname — 2026-11-05 (Derek), #1088. - It keeps its pair's reading where the clause is nothing but title modifiers and styles (`(UK)`, `(Hon)`), a lone title of address (`(Sir)`, `(Mr)`), titles alone straight behind a credential (`FRCS (SN)`), or a credential written as initials before a name word of its comma part (`Justin (JP) Stearns`); content that is not such a clause and ends in an ASCII period keeps v1's escape, its brackets dropped — 2026-11-04 (Derek), #792. - (superseded for a clause of non-name words by #792, 2026-11-04 below) The rule says what the escape does: quotes escape as brackets do, and a clause of two or more words is not suffix-shaped unless an ASCII period ends it, so `(PhD MSc)` keeps its nickname reading with nothing reported, as 1.4.0 read `(PhD MBA)`, which reports suffix-or-nickname since `mba` was marked ambiguous (decisions/suffix-acronym-collisions, 2026-11-03) — 2026-10-21 #807. @@ -18,3 +19,6 @@ Elsewhere: an ambiguous member alone in the brackets staying a nickname, reporte A REPORT THE COMMA FIX REACHES: a member of the ambiguous class written with the clause's comma is a tagged member now, so a nickname clause holding one reports `suffix-or-nickname`, as S13 reports any member left in a nickname (ORCID's `Prof Anh My Chu (BEng, MEng, DEng)` and `HIKONO, Masaru (MEng, Ph.D. in Human Science, CPE)`, whose `MEng,` had carried no tag). ACCEPTED COSTS (rules/S1's Accepted bullets): a clause between name words reads otherwise than the same words bare (`W. Russell Jr. Withers` keeps middle 'Russell Jr.'); a short title word a writer's data uses as a tag reads as the title (FEC's state codes, `CASHION, JOHN(ST)` and six more entries holding `(ST)`, two holding `(ME)`, by the same recipe); a title-only clause behind a credential keeps its pair's reading even where it holds a title of the name and loses the period escape it took (`John Smith, MD (Dr.)` → nickname 'Dr.'; `Louise Dye, PhD (Professor)` stays the nickname it was); a set-apart clause in a name of no word of its own leaves the name empty (`(MPH)` → suffix 'MPH', where the escape had made it given 'MPH', H4); and `Col. (Ret.) Smith` reads suffix 'Ret.' where 1.4.0 through 2.8.0 read title 'Col. Ret.' (tests/test_suffixes.py's two "known limitation" rows, `Lon (Jr.) Williams` now suffix 'Jr.', are re-pinned). MEASURED 2026-11-04 with `snapshot.py --local --base ` (head package digest 9b70692c0bafee65): ctgov-officials 26 entries move, fec-individuals 16, psc-individuals 10, orcid-researchers 79; govuk-people, ukec-individuals, cz-candidates and dof-signatures none. By class: a bracketed title read as the title where it read as the nickname (`Dorothee Auer (Principal Investigator)`, `Prof (Dr) Ashish Gadekar` → title 'Prof Dr', `Mary (Rev Mrs) Fosu`; FEC's state codes `(ST)` and `(ME)`, the accepted cost); a credential run read as the suffix where it read as the nickname (`Brian Chirombo (MBChB, MPH)`, `F.L.J. Visseren, Professor (MD PhD)`); a bracketed generation or credential between name words ending at its brackets (`Mr Hugh (Snr) Boyle` → suffix 'Snr' where middle, `Abebe Alemu (PhD) Balcha` → family 'Balcha', `Dr (Phd) Alan Green` → given 'Alan'); credential initials inside a name moving from the middle name or the suffix run to the nickname (`FOX, (JP)JIMMY`); two ORCID nickname clauses gaining the member report above; and FEC's `TOBIN, MARY (V)` losing the suffix-or-name report #877 gives a bare letter ending the given part after a comma, which master gave it because its extract stage drops the brackets from content that is suffix-shaped, reading it bare: the clause is set apart in post_rules, after assign's walk reports, and a letter its writer bracketed is set apart from the name rather than a middle initial in doubt, so rules/C9 names it among the slots that settle the letter (the first measurement, against c6207549 before #877 merged, counted FEC 16). CLDR agreement: one Czech rendering moves, `Machová, Alexandra Zuzana (paní, Ph.D.)`, its given names agreeing now and its title gaining 'paní', which CLDR leaves out. +- 2026-11-05 (Derek), #1088 — A JOINED CREDENTIAL IS SET APART AS ITS SPLIT SPELLING IS. `John Smith (MD, PhD)` read suffix 'MD, PhD' under #792, but `John Smith (MD/PhD)`, `John (MD/PhD) Smith` and `John Smith (MD-PhD)` read as nicknames, though S3 reads the bare `John Smith MD/PhD` as the suffix: the set-apart word asked its own copy of the unambiguous suffix sets, which S3's joiner arm is not in. #1084's simplify pass proposed asking `suffix_as_written`, the bare word's question, and refused it as a behavior change, since that predicate also reads a dotted member of the ambiguous class (`M.A.`) as a suffix, which S13 settles only after the name. DECIDED (Derek), as proposed: the word asks the bare word's question, a member of the ambiguous class asked first and left to S13, whose position test is unchanged. The member's dotted test, which post_rules had copied twice, is the word test's now, so the report on a member left the nickname leaves off a comma the clause keeps, as the settling did (rules/N1): `Jean (M.A,) Smith` reports suffix-or-nickname, as `Jean (M.A) Smith` does. The interior-period half of S3 (`(Msc.Ed)`, a different predicate whose any-chunk test is looser than the joiner's every-chunk) was left to an issue rather than folded in. + WHAT IT MOVES, and accepted: a joined credential ending in a period is set apart before the period escape is asked, which had read it bare, so `John (M.D./Ph.D.) Smith` → suffix where 1.4.0 through 2.8 read middle 'M.D./Ph.D.', and `Jack (M.D./Ph.D.)` → family 'Jack', as `Jack (PhD)` reads, where 2.8 read given 'Jack' and reported given-or-family. Not moved: a joined word with a chunk that is no credential or is a bare member (`MD/Bob`, `MA/PhD`), a hyphenated name (`Smith-Jones`, `Ae-Ra`), single-letter numerals (`I-V`) and a dangling joiner (`MD/`), each as S3 already refuses it bare. + MEASURED 2026-11-05 against master 4109b026 checked out in a detached worktree: the committed snapshot moves 0 of its 6,939 names, none of the 239 holding a delimiter having a joined word inside it, so the zero says the snapshot cannot see the shape. A grid of 30 clause words (thirteen that move: the joined credentials `MD/PhD`, `MD-PhD`, `Md-PhD`, `md/phd`, `MD/PhD/MPH`, `M.A./PhD`, `Jr/Sr`, `MD/PhD,`, `M.D./Ph.D.`, and the clauses `MD/PhD, MBA` and `PhD, MD/MPH` holding one, and the kept-comma members `M.A,` and `M.A.,`; seventeen that do not: S3's refusals bare, a bare member as a chunk (`PhD/MBA`, `MA/PhD`, `JD/MBA`), a chunk not written with its periods (`M.A/PhD`), an unlisted chunk (`RN-BC`, `CPA/PFS`, `XY/ZW`), a name or a title as a chunk (`MD/Bob`, `Bob/MD`, `Dr/PhD`), a dangling joiner (`MD/`, `/PhD`), single-letter numerals (`I-V`) and a hyphenated name (`Smith-Jones`, `Ae-Ra`), and the members without a comma, `M.A` and `MA`) in 14 shapes (after the name, inside it, before it, alone, after and before a family comma, before and after a suffix comma, a name in capitals, a lone name word, an empty head before a comma, before a suffix comma with nothing ahead, in quotes, and after a two-word Korean name), each under GIVEN_FIRST, FAMILY_FIRST and FAMILY_FIRST_GIVEN_LAST, 1,260 names: 447 move, 149 under each order, every move a joined credential read as the suffix or the member report. The comparator is the base tree's own readings of the same grid, parsed with each tree's `nameparser.__file__` asserted. In the local corpora (`snapshot.py --local --base `): ctgov-officials 3 entries move, `Franck Pages, Professor (MD-PHD)` and `Franck Pagès, Professor (MD-PHD)` and `Philippe PC Courtet, Professor(MD-PhD)` → suffix where nickname; fec-individuals 1, `FRANTZ (CAPT USN-RET), JOSEPH` → title 'CAPT', suffix 'USN-RET' where nickname 'CAPT USN-RET'; psc-individuals and orcid-researchers none. diff --git a/docs/design/decisions/S13-bracketed-member-settled-by-its-writing.md b/docs/design/decisions/S13-bracketed-member-settled-by-its-writing.md index b3e4f0817..31ed93087 100644 --- a/docs/design/decisions/S13-bracketed-member-settled-by-its-writing.md +++ b/docs/design/decisions/S13-bracketed-member-settled-by-its-writing.md @@ -24,3 +24,4 @@ Elsewhere: P6's attachment after a comma with nothing before it, which the #810 - **Accepted:** the twin differences the declined bullet above lists, in rules/S13's Accepted bullets. - **Measured** (2026-10-26, against master 12384fae with `--base` on a clean checkout, and again on the one-pass tree with a name word required before the clause, 482691a6, against master 13539131, #897, with the same moves): the committed snapshot moves 3 names, each a member read as the suffix in place of the nickname, its `suffix-or-nickname` report gone (`John Smith (MA)`, `Andrew Perkins (JD)`, `John Smith PhD (MA)` → suffix 'PhD MA'); every CLDR, GND, Wikidata and conventions agreement record matches, the locale views included. Of the local corpora, ctgov-officials moves 3 of the 29 names holding one member alone in brackets or quotes (the other 26: 2 already the suffix by a final period, `Ed.`; 23 in Title case, every one `Ed`, 3 of them inside the name and 1 in a one-case name; and 1 in lower case), each a capitals member in a mixed-case name after a credential behind a comma (shapes `Aa A Aa, AA (AA)`, `Aa Aa, AaAaAa (AA)`, `Aa Aa, Aa.AA(AA)`), read suffix, the credential it is. fec-individuals holds 6 such names, all in one-case names, and none moves; tse-donors and ukec-individuals hold none and match; psc-individuals holds none, and its agreement record matches. - **Recompute:** `uv run python tools/differential/snapshot.py` and `cldr_agreement.py` from this tree, `snapshot.py --local --base ` for ctgov-officials, fec-individuals, tse-donors and ukec-individuals, and `local_agreement.py psc-individuals --base `; the population is the names in each corpus holding a delimited clause of one word whose period-free fold is in `Lexicon.default().suffix_acronyms_ambiguous`, by delimiter pair, position, case and spelling. +- 2026-11-05 (Derek), #1088 — the report on a member left the nickname leaves off a comma the clause keeps on it, as the settling does: `Jean (M.A,) Smith` reports suffix-or-nickname. The member's dotted test is now the set-apart word's own, asked before the bare suffix question (decisions/S1's 2026-11-05 entry). diff --git a/docs/design/rules/S1-bracketed-credential-is-not-a-nickname.md b/docs/design/rules/S1-bracketed-credential-is-not-a-nickname.md index 41b5b9a99..91e7336e5 100644 --- a/docs/design/rules/S1-bracketed-credential-is-not-a-nickname.md +++ b/docs/design/rules/S1-bracketed-credential-is-not-a-nickname.md @@ -4,7 +4,7 @@ **Then:** a clause wholly of words the vocabulary reads as no name is set apart from the name: it is no nickname and no maiden name, and the words around it read as they would beside a nickname. -- **Each word takes the role its vocabulary gives it**: a title is the title, a credential or a generation the suffix, and a word that is both is the title in front of the name's first word, or in a name with no word of its own, and the suffix anywhere else. The vocabulary is asked of each word on its own, so a credential whose word is also a name (S4) is no word of such a clause, and neither is a title that is also a given name (H9). +- **Each word takes the role its vocabulary gives it**: a title is the title, a credential or a generation the suffix, one joined by a slash or a hyphen (S3) too, and a word that is both is the title in front of the name's first word, or in a name with no word of its own, and the suffix anywhere else. The vocabulary is asked of each word on its own, so a credential whose word is also a name (S4) is no word of such a clause, and neither is a title that is also a given name (H9). ``` "John Smith (PhD, MBA)" → suffix="PhD, MBA" "John Smith (Dr)" → title="Dr" @@ -13,7 +13,12 @@ "(Lt)" → title="Lt" "John (Sr.) Smith" → suffix="Sr." "John Smith (Mr, PhD)" → title="Mr" + "John Smith (MD/PhD)" → suffix="MD/PhD" + "John (MD-PhD) Smith" → suffix="MD-PhD" + "John (M.D./Ph.D.) Smith" → suffix="M.D./Ph.D." "John Smith (King)" → nickname="King" · boundary + "John Smith (MD/Bob)" → nickname="MD/Bob" · boundary + "John Smith (MA/PhD)" → nickname="MA/PhD" · boundary ``` - **The words around it read as they would beside a nickname**, so a clause between name words ends where its brackets do and takes neither neighbour, and a lone name word beside it is the family name (N3). ``` diff --git a/docs/design/rules/S13-bracketed-member-settled-by-its-writing.md b/docs/design/rules/S13-bracketed-member-settled-by-its-writing.md index e72f04aae..560c60e96 100644 --- a/docs/design/rules/S13-bracketed-member-settled-by-its-writing.md +++ b/docs/design/rules/S13-bracketed-member-settled-by-its-writing.md @@ -36,13 +36,14 @@ "john smith (MA)" → nickname="MA" "John Smith (Ma)" → nickname="Ma" ``` -- **A member standing inside the name or before it** stays the nickname, however it is written, and so does one in a name with no name word of its own, or one with a word after it the parse reads as a name word, a stray full stop included. +- **A member standing inside the name or before it** stays the nickname, however it is written, and so does one in a name with no name word of its own, or one with a word after it the parse reads as a name word, a stray full stop included. It is reported, a comma the clause keeps on it no part of it (N1). ``` "Jeffrey (JD) Bricken" → nickname="JD" "JEFFREY (JD) BRICKEN" → nickname="JD" "Jean 'JD' Smith" → nickname="JD" "Jean (M.A) Smith" → nickname="M.A" "Jean (M.A) Smith" → ambiguities=("suffix-or-nickname",) + "Jean (M.A,) Smith" → ambiguities=("suffix-or-nickname",) "Smith (MA), John" → nickname="MA" "(MA) John Smith" → nickname="MA" "Dr. (MA)" → nickname="MA" diff --git a/nameparser/_pipeline/_extract.py b/nameparser/_pipeline/_extract.py index d9a2e18c2..c7902c58f 100644 --- a/nameparser/_pipeline/_extract.py +++ b/nameparser/_pipeline/_extract.py @@ -71,7 +71,8 @@ def _set_apart(text: str, inner: Span, kinds = [] for word in words: kind = set_apart_kind(word, lexicon) - if kind is None: + # a member settled by its writing is weighed in post_rules + if kind is None or kind == "member": return False kinds.append(kind) return bool(words) and set_apart(words, kinds, False, lexicon) diff --git a/nameparser/_pipeline/_post_rules.py b/nameparser/_pipeline/_post_rules.py index 1b014bade..dae8a423c 100644 --- a/nameparser/_pipeline/_post_rules.py +++ b/nameparser/_pipeline/_post_rules.py @@ -689,15 +689,12 @@ def _set_apart_clauses(state: ParseState, for i in range(lo, hi): token = tokens[i] kind = set_apart_kind(token.text, lex) + if kind is None and AMBIGUOUS_ACRONYM_TAG in token.tags \ + and settled_by_case(token, state.one_case): + kind = "member" if kind is None: - bare = token.text.strip(",") - dotted = ("." in bare.strip(".") - and _normalize(bare).replace(".", "") - in lex._sets.suffix_acronyms_ambiguous) - if not (dotted or (AMBIGUOUS_ACRONYM_TAG in token.tags - and settled_by_case(token, - state.one_case))): - break + break + if kind == "member": kind, member = "suffix", True kinds.append(kind) else: @@ -775,16 +772,18 @@ def _report_delimited_members(tokens: list[WorkToken], lexicon: Lexicon, included. Reported here, where the clause is decided, rather than in extract, which has no token index to point at. A dotted member carries no tag in a clause (classify tags a shape only outside - one), so it is found by its text.""" + one), so it is found by its text, through `set_apart_kind` as the + settling finds it.""" for i, token in enumerate(tokens): if token.role is not Role.NICKNAME: continue - # a period inside the word ('M.A'); one ending it is extract's - # escape - dotted = ("." in token.text.strip(".") - and _normalize(token.text).replace(".", "") - in lexicon._sets.suffix_acronyms_ambiguous) - if dotted or AMBIGUOUS_ACRONYM_TAG in token.tags: + # a period inside the word ('M.A'), a comma the clause kept on + # it left off; one ending it is extract's escape. The period is + # asked first, at C level, so a nickname written without one + # pays no frame + if (AMBIGUOUS_ACRONYM_TAG in token.tags + or "." in token.text + and set_apart_kind(token.text, lexicon) == "member"): ambiguities.append(PendingAmbiguity( AmbiguityKind.SUFFIX_OR_NICKNAME, f"delimited {token.text!r} is also a post-nominal; read " diff --git a/nameparser/_pipeline/_vocab.py b/nameparser/_pipeline/_vocab.py index a60212a76..6a83d9e52 100644 --- a/nameparser/_pipeline/_vocab.py +++ b/nameparser/_pipeline/_vocab.py @@ -400,21 +400,28 @@ def ends_in_a_generation(texts: Sequence[str], one_case: bool | None, # rules/S1: "a clause wholly of words the vocabulary reads as no name # is set apart from the name" -- the word's half; `set_apart` is the -# clause's (decisions/S1, #792) +# clause's. Its suffix half is `suffix_as_written`'s, so a credential +# joined by a slash or a hyphen is one here as it is bare, a member of +# the ambiguous class asked first and left to S13's position test +# (decisions/S1, #792, #1088) def set_apart_kind(word: str, lexicon: Lexicon, - ) -> Literal["title", "suffix", "dual"] | None: + ) -> Literal["title", "suffix", "dual", "member"] | None: """What a word in a delimited clause is to S1's set-apart reading: - a title, a suffix (an unambiguous suffix word or acronym), both, or - None where the vocabulary reads it as a name or not at all. A comma - the clause keeps on its word is no part of it (rules/N1).""" - folded = _normalize(word.strip(",")) + a title, a suffix (a word `suffix_as_written` reads, joined + credentials included), both, a member of the ambiguous class + written with a period inside it, which S13 settles after the name + only, or None where the vocabulary reads it as a name or not at + all. A comma the clause keeps on its word is no part of it + (rules/N1).""" + bare = word.strip(",") + folded = _normalize(bare) sets = lexicon._sets - acronym = folded.replace(".", "") - suffix = ((folded in sets.suffix_words - and folded not in sets.suffix_words_ambiguous) - or (acronym in sets.suffix_acronyms - and acronym not in sets.suffix_acronyms_ambiguous)) title = folded in sets.titles + if folded.replace(".", "") in sets.suffix_acronyms_ambiguous: + if title: + return "title" + return "member" if "." in bare.strip(".") else None + suffix = suffix_as_written(folded, bare, lexicon) if title: return "dual" if suffix else "title" return "suffix" if suffix else None diff --git a/tests/v2/cases.py b/tests/v2/cases.py index 3ede207a5..f775ed726 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -9797,6 +9797,51 @@ def _check_cjk_shape_purity(self) -> None: notes="a comma opening a word is trimmed off its front, so the " "word's span starts after it (rules/S1); 1.4.0 and 2.8 " "read nickname 'MD ,PhD', 2.0.0 'MD PhD'"), + # rules/S1 (#1088): a credential joined by a slash or a hyphen, which + # S3 reads as a suffix bare, is set apart in brackets as its split + # spelling '(MD, PhD)' is + Case("paren_slash_joined_credential_set_apart", "John Smith (MD/PhD)", + {"given": "John", "family": "Smith", "suffix": "MD/PhD"}, + classification="fix(#1088)", + notes="bare 'John Smith MD/PhD' reads suffix (rules/S3); 1.4.0 " + "through 2.8 read nickname 'MD/PhD'"), + Case("paren_hyphen_joined_credential_inside_the_name_set_apart", + "John (MD-PhD) Smith", + {"given": "John", "family": "Smith", "suffix": "MD-PhD"}, + classification="fix(#1088)", + notes="no initials' shape, so S1's initials Except does not " + "reach it; 1.4.0 through 2.8 read nickname 'MD-PhD'"), + Case("paren_joined_credential_before_a_family_comma_set_apart", + "Smith (MD/PhD), John", + {"given": "John", "family": "Smith", "suffix": "MD/PhD"}, + classification="fix(#1088)", + notes="as 'Smith (MD), John' reads; 1.4.0 through 2.8 read " + "nickname 'MD/PhD'"), + Case("paren_dotted_joined_credential_set_apart_not_escaped", + "John (M.D./Ph.D.) Smith", + {"given": "John", "family": "Smith", "suffix": "M.D./Ph.D."}, + classification="fix(#1088)", + notes="set apart before the period escape is asked, which had " + "read it bare: 1.4.0 through 2.8 read middle " + "'M.D./Ph.D.'"), + Case("paren_dotted_joined_credential_after_a_lone_name_word", + "Jack (M.D./Ph.D.)", + {"family": "Jack", "suffix": "M.D./Ph.D."}, + classification="fix(#1088)", + notes="the lone word is N3's family name, as in 'Jack (PhD)'; " + "2.8 read given 'Jack', reporting given-or-family, and " + "1.4.0 last 'M.D./Ph.D.'"), + Case("paren_joined_credential_and_name_stays_the_nickname", + "John Smith (MD/Bob)", + {"given": "John", "family": "Smith", "nickname": "MD/Bob"}, + notes="S3 asks every chunk, and Bob is no credential"), + Case("paren_joined_member_chunk_stays_the_nickname", + "John Smith (MA/PhD)", + {"given": "John", "family": "Smith", "nickname": "MA/PhD"}, + notes="a bare member of the ambiguous class is no chunk of a " + "joined credential (rules/S3), so the clause is not " + "set apart; the joined word is no member either, so " + "nothing reports"), # rules/S13 (#810): a member of the ambiguous class alone in the # brackets after the name, settled by its writing. Each row is # paired with a control its writing or its position does not settle. @@ -9859,6 +9904,14 @@ def _check_cjk_shape_purity(self) -> None: ambiguities=("suffix-or-nickname",), notes="inside the name however it is written, and reported as " "a capitals member is; 1.4.0 read middle 'M.A'"), + Case("bracket_member_with_a_kept_comma_reports", + "Jean (M.A,) Smith", + {"given": "Jean", "family": "Smith", "nickname": "M.A,"}, + classification="fix(#1088)", + ambiguities=("suffix-or-nickname",), + notes="the comma the clause keeps is no part of the member " + "(rules/N1), so it reports as 'Jean (M.A) Smith' does; " + "2.8 reported nothing, and 1.4.0 read nickname 'M.A,'"), Case("bracket_member_before_a_family_comma_stays_the_nickname", "Smith (MA), John", {"given": "John", "family": "Smith", "nickname": "MA"}, diff --git a/tests/v2/pipeline/test_post_rules.py b/tests/v2/pipeline/test_post_rules.py index 87b9fb64a..f7602864e 100644 --- a/tests/v2/pipeline/test_post_rules.py +++ b/tests/v2/pipeline/test_post_rules.py @@ -7,10 +7,11 @@ import pytest from nameparser import Locale, Parser, PolicyPatch, locales, parse, parser_for -from nameparser._lexicon import Lexicon +from nameparser._lexicon import Lexicon, _normalize from nameparser._pipeline import run from nameparser._pipeline import _post_rules from nameparser._pipeline._post_rules import suffix_entries +from nameparser._pipeline._vocab import suffix_as_written from nameparser._pipeline._state import ParseState from nameparser._policy import (FAMILY_FIRST, FAMILY_FIRST_GIVEN_LAST, GIVEN_FIRST, PatronymicRule, Policy, @@ -1767,3 +1768,57 @@ def test_a_set_apart_clause_trims_its_comma_off_the_word(text: str) -> None: for t in name.tokens if t.span is not None) # and every row is one the trim moves assert _SET_APART_COMMAS[text] != _UNGUARDED_EFFECT[text] + + +#: rules/S1's set-apart word asks `suffix_as_written`, the question a +#: bare word is asked, so a credential S3 reads joined by a slash or a +#: hyphen is set apart in brackets as its split spelling is (#1088): +#: each word bracketed after the name and inside it. +_BRACKETED_WORDS = ("PhD", "Jr.", "MD/PhD", "MD-PhD", "md/phd", "Md-PhD", + "MD/PhD/MPH", "M.D./Ph.D.", "M.A./PhD", "Jr/Sr", + "MD/Bob", "MA/PhD", "Smith-Jones", "Ae-Ra", "I-V", "MD/") + +#: The rows the guard moves, as the tree before #1088 read them, the +#: set-apart word asking its own copy of the unambiguous suffix sets: +#: the test's negative control, recorded 2026-11-05 from a detached +#: worktree of 4109b026 as (suffix, nickname). The joined words were +#: nicknames, and the dotted one inside the name took the period escape +#: and read as the middle name. +_UNGUARDED_EFFECT_1088 = { + "John Smith (MD/PhD)": ("", "MD/PhD"), + "John (MD/PhD) Smith": ("", "MD/PhD"), + "John Smith (MD-PhD)": ("", "MD-PhD"), + "John (MD-PhD) Smith": ("", "MD-PhD"), + "John Smith (md/phd)": ("", "md/phd"), + "John (md/phd) Smith": ("", "md/phd"), + "John Smith (Md-PhD)": ("", "Md-PhD"), + "John (Md-PhD) Smith": ("", "Md-PhD"), + "John Smith (MD/PhD/MPH)": ("", "MD/PhD/MPH"), + "John (MD/PhD/MPH) Smith": ("", "MD/PhD/MPH"), + "John (M.D./Ph.D.) Smith": ("", ""), + "John Smith (M.A./PhD)": ("", "M.A./PhD"), + "John (M.A./PhD) Smith": ("", "M.A./PhD"), + "John Smith (Jr/Sr)": ("", "Jr/Sr"), + "John (Jr/Sr) Smith": ("", "Jr/Sr"), +} + + +@pytest.mark.parametrize("word", _BRACKETED_WORDS) +def test_a_bracketed_word_is_the_suffix_where_the_bare_word_is_one( + word: str) -> None: + lexicon = Lexicon.default() + suffix = suffix_as_written(_normalize(word), word, lexicon) + for text in (f"John Smith ({word})", f"John ({word}) Smith"): + name = parse(text) + read = (name.suffix, name.nickname) + assert read == ((word, "") if suffix else ("", word)), text + if text in _UNGUARDED_EFFECT_1088: + assert _UNGUARDED_EFFECT_1088[text] != read + + +def test_the_bracketed_word_control_names_rows_of_the_grid() -> None: + # a control row outside the grid would guard nothing + texts = {t for w in _BRACKETED_WORDS + for t in (f"John Smith ({w})", f"John ({w}) Smith")} + assert set(_UNGUARDED_EFFECT_1088) <= texts + assert len(_UNGUARDED_EFFECT_1088) == 15 diff --git a/tests/v2/rules_budget.json b/tests/v2/rules_budget.json index 9da0bf841..a0c549f2b 100644 --- a/tests/v2/rules_budget.json +++ b/tests/v2/rules_budget.json @@ -64,7 +64,7 @@ "R9": {"words": 397, "then_words": 122, "because_sentences": 3, "except": 5, "precedence": 0, "accepted": 13, "accepted_words": 477}, "R10": {"words": 168, "then_words": 26, "because_sentences": 2, "except": 2, "precedence": 0, "accepted": 6, "accepted_words": 160}, "R11": {"words": 226, "then_words": 42, "because_sentences": 3, "except": 2, "precedence": 0, "accepted": 5, "accepted_words": 195}, - "S1": {"words": 704, "then_words": 238, "because_sentences": 3, "except": 6, "precedence": 3, "accepted": 5, "accepted_words": 173}, + "S1": {"words": 714, "then_words": 248, "because_sentences": 3, "except": 6, "precedence": 3, "accepted": 5, "accepted_words": 173}, "S2": {"words": 658, "then_words": 120, "because_sentences": 1, "except": 0, "precedence": 7, "accepted": 3, "accepted_words": 222}, "S3": {"words": 960, "then_words": 398, "because_sentences": 3, "except": 4, "precedence": 5, "accepted": 6, "accepted_words": 357}, "S4": {"words": 306, "then_words": 164, "because_sentences": 2, "except": 0, "precedence": 3, "accepted": 0, "accepted_words": 0}, @@ -76,7 +76,7 @@ "S10": {"words": 721, "then_words": 246, "because_sentences": 3, "except": 4, "precedence": 3, "accepted": 3, "accepted_words": 163}, "S11": {"words": 745, "then_words": 466, "because_sentences": 3, "except": 1, "precedence": 1, "accepted": 4, "accepted_words": 247}, "S12": {"words": 766, "then_words": 372, "because_sentences": 2, "except": 3, "precedence": 4, "accepted": 2, "accepted_words": 93}, - "S13": {"words": 433, "then_words": 182, "because_sentences": 3, "except": 2, "precedence": 1, "accepted": 1, "accepted_words": 46}, + "S13": {"words": 448, "then_words": 182, "because_sentences": 3, "except": 2, "precedence": 1, "accepted": 1, "accepted_words": 46}, "T1": {"words": 99, "then_words": 18, "because_sentences": 1, "except": 1, "precedence": 0, "accepted": 3, "accepted_words": 91}, "T2": {"words": 43, "then_words": 14, "because_sentences": 1, "except": 0, "precedence": 0, "accepted": 0, "accepted_words": 0}, "T3": {"words": 56, "then_words": 23, "because_sentences": 1, "except": 0, "precedence": 0, "accepted": 0, "accepted_words": 0}, diff --git a/tests/v2/test_cases.py b/tests/v2/test_cases.py index e55b3779c..25babe8a3 100644 --- a/tests/v2/test_cases.py +++ b/tests/v2/test_cases.py @@ -362,9 +362,11 @@ def test_the_field_sweep_sees_the_claims_it_checks() -> None: # particle no longer report particle-or-given under the two # family-first orders, the fold having taken it; 1597 -> 1606 on # 2026-11-05, #1080's two boundary rows and its paired-initials row, - # each reporting suffix-or-name under three orders. - assert claims == 1606, ( - f"the field sweep checks {claims} claims, recorded as 1606 on " + # each reporting suffix-or-name under three orders; 1606 -> 1609 on + # 2026-11-05, #1088's 'Jean (M.A,) Smith' row reporting + # suffix-or-nickname under three orders. + assert claims == 1609, ( + f"the field sweep checks {claims} claims, recorded as 1609 on " f"2026-11-05") diff --git a/tests/v2/test_layering.py b/tests/v2/test_layering.py index 437c2cfb2..ac586a36b 100644 --- a/tests/v2/test_layering.py +++ b/tests/v2/test_layering.py @@ -648,7 +648,10 @@ def test_the_check_finds_a_tool_read_through_an_alias( #: `title_case_name_words`, no 2.x field either; #694's phrase gate #: one load, `bound_given_names`; #792's set-apart clause two `sets =` #: bindings and nine loads in `_vocab` and two loads in `_post_rules`, -#: taking extract's four suffix-set loads out, 119 -> 128); the tools' 19 are 17 +#: taking extract's four suffix-set loads out, 119 -> 128; #1088's +#: set-apart word asking `suffix_as_written`, three loads out of +#: `_vocab` and both of `_post_rules`' copies of the dotted member +#: test, 128 -> 123); the tools' 19 are 17 #: attribute loads, every tool scanned, and config_diff.py's #: `sets =` binding and the roster `getattr` through it (its #: `changed_words`, which compares two lexicons set by set), which @@ -658,7 +661,7 @@ def test_the_check_finds_a_tool_read_through_an_alias( #: `_lexicon._fields_of` from 4i-b until 4i-c retired it. #: A reader added through `_sets` moves these, and they are re-taken #: then: they count what the check would refuse with the move undone. -_UNGUARDED_READS = {"library": 128, "tools": 19} +_UNGUARDED_READS = {"library": 123, "tools": 19} def test_the_check_refuses_every_read_4i_a_moved() -> None: From 867bb9528813cbf082459c2c3d550d5b1996777f Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Thu, 5 Nov 2026 16:21:18 -0800 Subject: [PATCH 2/4] Read a joined credential as its words written apart; review fixes (#1088) Parity, as decided: rules/S1's initials Except asks each chunk of a credential joined by a slash or a hyphen (`_vocab.reads_as_initials`), so `John (MD-PhD) Smith` and `Justin (JP/MD) Stearns` stay the nickname before a name word, as `(MD, PhD)` and `(JP, MD)` do, while `John (Jr/Sr) Smith` reads suffix as `(Jr, Sr)` does. A two-spelling guard holds the pair alike over 2,310 rows (540 apart with the chunk loop off). Review fixes: `_vocab.dotted_member` is S13's one dotted test, asked by the settling and the report, so a caller listing `m.a` as a title keeps the report; `SetApartKind` names the kind set; new case rows for every comma shape, a family-first order, the lost `(MD/PhD, MBA)` report, the kept-comma members and the period escape extract keeps for a dotted member; the guard pins S3's refusals apart from its oracle; caller-lexicon tests for the title-and-member paths. decisions/S1 re-counts the snapshot (6,945 entries, 6,736 distinct; 181 with a delimiter pair) with its recipe; corpus_rules.jsonl and snapshot.jsonl gain the nine new rule examples, no existing entry moving. Co-Authored-By: Claude Opus 5.5 --- ...-bracketed-credential-is-not-a-nickname.md | 9 +- ...-bracketed-credential-is-not-a-nickname.md | 9 +- ...bracketed-member-settled-by-its-writing.md | 2 +- nameparser/_pipeline/_extract.py | 1 - nameparser/_pipeline/_post_rules.py | 37 ++--- nameparser/_pipeline/_vocab.py | 79 ++++++++-- tests/v2/cases.py | 108 +++++++++++-- tests/v2/pipeline/test_post_rules.py | 143 ++++++++++++++---- tests/v2/rules_budget.json | 2 +- tests/v2/test_cases.py | 7 +- tests/v2/test_layering.py | 6 +- tests/v2/test_regex_sync.py | 6 +- tools/differential/corpus_rules.jsonl | 9 ++ tools/differential/snapshot.jsonl | 9 ++ 14 files changed, 328 insertions(+), 99 deletions(-) diff --git a/docs/design/decisions/S1-bracketed-credential-is-not-a-nickname.md b/docs/design/decisions/S1-bracketed-credential-is-not-a-nickname.md index 17c5c20ee..a90eb7386 100644 --- a/docs/design/decisions/S1-bracketed-credential-is-not-a-nickname.md +++ b/docs/design/decisions/S1-bracketed-credential-is-not-a-nickname.md @@ -3,7 +3,7 @@ ## Standing - A clause wholly of words the vocabulary reads as no name is set apart from the name: it is no nickname and no maiden name, each word takes the role its vocabulary gives it (a title the title, a credential or generation the suffix, a word that is both the title in front of the name's first word and the suffix elsewhere), the words around it read as they would beside a nickname (a lone name word beside it the family name, N3), and a member of the ambiguous class is one of its words where its writing settles it after the name (S13): `John Smith (PhD, MBA)` and `John Smith (Ph.D., MBA)` → suffix, `John Smith (Dr)` → title, `John Smith (PhD) Withers` → family 'Withers', suffix 'PhD' — 2026-11-04 (Derek), #792. -- A credential or generation S3 reads joined by a slash or a hyphen is a word of such a clause, asked as the bare word is asked, a member of the ambiguous class first left to S13: `John Smith (MD/PhD)`, `John (MD-PhD) Smith` → suffix, `John Smith (MD/Bob)` and `(MA/PhD)` → nickname — 2026-11-05 (Derek), #1088. +- A credential or generation S3 reads joined by a slash or a hyphen is a word of such a clause, asked as the bare word is asked, a member of the ambiguous class first left to S13, and the initials Except asks each of its chunks, so it reads as its words written apart do: `John Smith (MD/PhD)` → suffix, `John (MD-PhD) Smith` and `Justin (JP/MD) Stearns` → nickname, as `(MD, PhD)` and `(JP, MD)` read there, `John Smith (MD/Bob)` and `(MA/PhD)` → nickname — 2026-11-05 (Derek), #1088. - It keeps its pair's reading where the clause is nothing but title modifiers and styles (`(UK)`, `(Hon)`), a lone title of address (`(Sir)`, `(Mr)`), titles alone straight behind a credential (`FRCS (SN)`), or a credential written as initials before a name word of its comma part (`Justin (JP) Stearns`); content that is not such a clause and ends in an ASCII period keeps v1's escape, its brackets dropped — 2026-11-04 (Derek), #792. - (superseded for a clause of non-name words by #792, 2026-11-04 below) The rule says what the escape does: quotes escape as brackets do, and a clause of two or more words is not suffix-shaped unless an ASCII period ends it, so `(PhD MSc)` keeps its nickname reading with nothing reported, as 1.4.0 read `(PhD MBA)`, which reports suffix-or-nickname since `mba` was marked ambiguous (decisions/suffix-acronym-collisions, 2026-11-03) — 2026-10-21 #807. @@ -19,6 +19,7 @@ Elsewhere: an ambiguous member alone in the brackets staying a nickname, reporte A REPORT THE COMMA FIX REACHES: a member of the ambiguous class written with the clause's comma is a tagged member now, so a nickname clause holding one reports `suffix-or-nickname`, as S13 reports any member left in a nickname (ORCID's `Prof Anh My Chu (BEng, MEng, DEng)` and `HIKONO, Masaru (MEng, Ph.D. in Human Science, CPE)`, whose `MEng,` had carried no tag). ACCEPTED COSTS (rules/S1's Accepted bullets): a clause between name words reads otherwise than the same words bare (`W. Russell Jr. Withers` keeps middle 'Russell Jr.'); a short title word a writer's data uses as a tag reads as the title (FEC's state codes, `CASHION, JOHN(ST)` and six more entries holding `(ST)`, two holding `(ME)`, by the same recipe); a title-only clause behind a credential keeps its pair's reading even where it holds a title of the name and loses the period escape it took (`John Smith, MD (Dr.)` → nickname 'Dr.'; `Louise Dye, PhD (Professor)` stays the nickname it was); a set-apart clause in a name of no word of its own leaves the name empty (`(MPH)` → suffix 'MPH', where the escape had made it given 'MPH', H4); and `Col. (Ret.) Smith` reads suffix 'Ret.' where 1.4.0 through 2.8.0 read title 'Col. Ret.' (tests/test_suffixes.py's two "known limitation" rows, `Lon (Jr.) Williams` now suffix 'Jr.', are re-pinned). MEASURED 2026-11-04 with `snapshot.py --local --base ` (head package digest 9b70692c0bafee65): ctgov-officials 26 entries move, fec-individuals 16, psc-individuals 10, orcid-researchers 79; govuk-people, ukec-individuals, cz-candidates and dof-signatures none. By class: a bracketed title read as the title where it read as the nickname (`Dorothee Auer (Principal Investigator)`, `Prof (Dr) Ashish Gadekar` → title 'Prof Dr', `Mary (Rev Mrs) Fosu`; FEC's state codes `(ST)` and `(ME)`, the accepted cost); a credential run read as the suffix where it read as the nickname (`Brian Chirombo (MBChB, MPH)`, `F.L.J. Visseren, Professor (MD PhD)`); a bracketed generation or credential between name words ending at its brackets (`Mr Hugh (Snr) Boyle` → suffix 'Snr' where middle, `Abebe Alemu (PhD) Balcha` → family 'Balcha', `Dr (Phd) Alan Green` → given 'Alan'); credential initials inside a name moving from the middle name or the suffix run to the nickname (`FOX, (JP)JIMMY`); two ORCID nickname clauses gaining the member report above; and FEC's `TOBIN, MARY (V)` losing the suffix-or-name report #877 gives a bare letter ending the given part after a comma, which master gave it because its extract stage drops the brackets from content that is suffix-shaped, reading it bare: the clause is set apart in post_rules, after assign's walk reports, and a letter its writer bracketed is set apart from the name rather than a middle initial in doubt, so rules/C9 names it among the slots that settle the letter (the first measurement, against c6207549 before #877 merged, counted FEC 16). CLDR agreement: one Czech rendering moves, `Machová, Alexandra Zuzana (paní, Ph.D.)`, its given names agreeing now and its title gaining 'paní', which CLDR leaves out. -- 2026-11-05 (Derek), #1088 — A JOINED CREDENTIAL IS SET APART AS ITS SPLIT SPELLING IS. `John Smith (MD, PhD)` read suffix 'MD, PhD' under #792, but `John Smith (MD/PhD)`, `John (MD/PhD) Smith` and `John Smith (MD-PhD)` read as nicknames, though S3 reads the bare `John Smith MD/PhD` as the suffix: the set-apart word asked its own copy of the unambiguous suffix sets, which S3's joiner arm is not in. #1084's simplify pass proposed asking `suffix_as_written`, the bare word's question, and refused it as a behavior change, since that predicate also reads a dotted member of the ambiguous class (`M.A.`) as a suffix, which S13 settles only after the name. DECIDED (Derek), as proposed: the word asks the bare word's question, a member of the ambiguous class asked first and left to S13, whose position test is unchanged. The member's dotted test, which post_rules had copied twice, is the word test's now, so the report on a member left the nickname leaves off a comma the clause keeps, as the settling did (rules/N1): `Jean (M.A,) Smith` reports suffix-or-nickname, as `Jean (M.A) Smith` does. The interior-period half of S3 (`(Msc.Ed)`, a different predicate whose any-chunk test is looser than the joiner's every-chunk) was left to an issue rather than folded in. - WHAT IT MOVES, and accepted: a joined credential ending in a period is set apart before the period escape is asked, which had read it bare, so `John (M.D./Ph.D.) Smith` → suffix where 1.4.0 through 2.8 read middle 'M.D./Ph.D.', and `Jack (M.D./Ph.D.)` → family 'Jack', as `Jack (PhD)` reads, where 2.8 read given 'Jack' and reported given-or-family. Not moved: a joined word with a chunk that is no credential or is a bare member (`MD/Bob`, `MA/PhD`), a hyphenated name (`Smith-Jones`, `Ae-Ra`), single-letter numerals (`I-V`) and a dangling joiner (`MD/`), each as S3 already refuses it bare. - MEASURED 2026-11-05 against master 4109b026 checked out in a detached worktree: the committed snapshot moves 0 of its 6,939 names, none of the 239 holding a delimiter having a joined word inside it, so the zero says the snapshot cannot see the shape. A grid of 30 clause words (thirteen that move: the joined credentials `MD/PhD`, `MD-PhD`, `Md-PhD`, `md/phd`, `MD/PhD/MPH`, `M.A./PhD`, `Jr/Sr`, `MD/PhD,`, `M.D./Ph.D.`, and the clauses `MD/PhD, MBA` and `PhD, MD/MPH` holding one, and the kept-comma members `M.A,` and `M.A.,`; seventeen that do not: S3's refusals bare, a bare member as a chunk (`PhD/MBA`, `MA/PhD`, `JD/MBA`), a chunk not written with its periods (`M.A/PhD`), an unlisted chunk (`RN-BC`, `CPA/PFS`, `XY/ZW`), a name or a title as a chunk (`MD/Bob`, `Bob/MD`, `Dr/PhD`), a dangling joiner (`MD/`, `/PhD`), single-letter numerals (`I-V`) and a hyphenated name (`Smith-Jones`, `Ae-Ra`), and the members without a comma, `M.A` and `MA`) in 14 shapes (after the name, inside it, before it, alone, after and before a family comma, before and after a suffix comma, a name in capitals, a lone name word, an empty head before a comma, before a suffix comma with nothing ahead, in quotes, and after a two-word Korean name), each under GIVEN_FIRST, FAMILY_FIRST and FAMILY_FIRST_GIVEN_LAST, 1,260 names: 447 move, 149 under each order, every move a joined credential read as the suffix or the member report. The comparator is the base tree's own readings of the same grid, parsed with each tree's `nameparser.__file__` asserted. In the local corpora (`snapshot.py --local --base `): ctgov-officials 3 entries move, `Franck Pages, Professor (MD-PHD)` and `Franck Pagès, Professor (MD-PHD)` and `Philippe PC Courtet, Professor(MD-PhD)` → suffix where nickname; fec-individuals 1, `FRANTZ (CAPT USN-RET), JOSEPH` → title 'CAPT', suffix 'USN-RET' where nickname 'CAPT USN-RET'; psc-individuals and orcid-researchers none. +- 2026-11-05 (Derek), #1088 — A JOINED CREDENTIAL READS AS ITS WORDS WRITTEN APART. `John Smith (MD, PhD)` read suffix 'MD, PhD' under #792, but `John Smith (MD/PhD)` and `John Smith (MD-PhD)` read as nicknames, though S3 reads the bare `John Smith MD/PhD` as the suffix: the set-apart word asked its own copy of the unambiguous suffix sets, which S3's joiner arm is not in. #1084's simplify pass proposed asking `suffix_as_written`, the bare word's question, and refused it as a behavior change, since that predicate also reads a dotted member of the ambiguous class (`M.A.`) as a suffix, which S13 settles only after the name. DECIDED (Derek), as proposed: the word asks the bare word's question, a member of the ambiguous class asked first and left to S13, whose position test is unchanged. The member's dotted test, which post_rules had copied twice, is one predicate now (`_vocab.dotted_member`), asked by the settling and by the report alike, so the report on a member left the nickname leaves off a comma the clause keeps, as the settling did (rules/N1): `Jean (M.A,) Smith` reports suffix-or-nickname, as `Jean (M.A) Smith` does. The interior-period half of S3 (`(Msc.Ed)`, a different predicate whose any-chunk test is looser than the joiner's every-chunk) was left out, for an issue drafted the same day. + THE INITIALS EXCEPT, decided after review: the first build set a joined credential apart before a name word too, where its words written apart stay the nickname under S1's initials Except, which asked the whole word, and a joined word never has the shape of initials (`John (MD/PhD) Smith` and `Justin (JP/MD) Stearns` → suffix, against `John (MD, PhD) Smith` and `Justin (JP, MD) Stearns` → nickname; the code, comment and design-docs reviews each found it, the last by the two-spelling invariant over 3,360 pairs). OPTIONS PUT: (A) the Except asks each chunk, so the two spellings read alike; (B) keep the divergence as an accepted cost. DECIDED (Derek): A. A joined credential counts as initials where any of its chunks would, as its words written apart do (`JP/MD`, `MD/PhD`, `M.D./Ph.D.`), and one whose chunks are generation words does not (`John (Jr/Sr) Smith` → suffix, as `(Jr, Sr)` reads). tests/v2/pipeline/test_post_rules.py holds the two spellings alike over a grid of plain credential pairs in eleven shapes under the three orders; with the chunk loop off, 540 of its 2,310 rows read apart. Where the spelling written apart reads by another rule the joined word does not meet, the two still differ, each by that rule: a word that is a title too reads as the title alone in a name with no word of its own (`(MD, MS)` → title 'MS', `(MD/MS)` → suffix 'MD/MS'), and a clause ending in a member written with its periods takes the period escape written apart (`John (MD, M.A.) Smith` → suffix 'MD, M.A. Smith', `John (MD/M.A.) Smith` → nickname). + WHAT IT MOVES, and accepted: a joined credential ending in a period is set apart before the period escape is asked, which had read it bare, so `Smith (M.D./Ph.D.), John` → suffix as 2.8 read it through the escape, `Jack (M.D./Ph.D.)` → family 'Jack', as `Jack (PhD)` reads, where 2.8 read given 'Jack' and reported given-or-family, and `John (M.D./Ph.D.) Smith`, which the initials Except declines, keeps its pair's reading rather than the escape (nickname, where 1.4.0 through 2.8 read middle 'M.D./Ph.D.'). A clause the joined word completes is set apart whole, so a member beside it that its capitals settle no longer reports (`John Smith (MD/PhD, MBA)` → suffix 'MD/PhD, MBA', where 2.8 read the nickname and reported suffix-or-nickname). Not moved: a joined word with a chunk that is no credential or is a bare member (`MD/Bob`, `MA/PhD`), a hyphenated name (`Smith-Jones`, `Ae-Ra`), single-letter numerals (`I-V`) and a dangling joiner (`MD/`), each as S3 already refuses it bare. + MEASURED 2026-11-05 against master 4109b026 checked out in a detached worktree: the committed snapshot moves none of its 6,945 entries (6,736 distinct names), and none of the 181 distinct names holding a pair of `DEFAULT_NICKNAME_DELIMITERS` has a word joined by a slash or a hyphen inside one, so the zero says the snapshot could not see the shape (recipe: for each name and each pair, the text between an opener and the next closer after it, matched against `\S[-/]\S`); the rule's nine new examples join the snapshot through corpus_rules.jsonl in the same change, recorded as new entries. A grid of 30 clause words (thirteen that move: the joined credentials `MD/PhD`, `MD-PhD`, `Md-PhD`, `md/phd`, `MD/PhD/MPH`, `M.A./PhD`, `Jr/Sr`, `MD/PhD,`, `M.D./Ph.D.`, and the clauses `MD/PhD, MBA` and `PhD, MD/MPH` holding one, and the kept-comma members `M.A,` and `M.A.,`; seventeen that do not: S3's refusals bare, a bare member as a chunk (`PhD/MBA`, `MA/PhD`, `JD/MBA`), a chunk not written with its periods (`M.A/PhD`), an unlisted chunk (`RN-BC`, `CPA/PFS`, `XY/ZW`), a name or a title as a chunk (`MD/Bob`, `Bob/MD`, `Dr/PhD`), a dangling joiner (`MD/`, `/PhD`), single-letter numerals (`I-V`) and a hyphenated name (`Smith-Jones`, `Ae-Ra`), and the members without a comma, `M.A` and `MA`) in 14 shapes (after the name, inside it, before it, alone, after and before a family comma, before and after a suffix comma, a name in capitals, a lone name word, an empty head before a comma, before a suffix comma with nothing ahead, in quotes, and after a two-word Korean name), each under GIVEN_FIRST, FAMILY_FIRST and FAMILY_FIRST_GIVEN_LAST, 1,260 names: 399 move, 133 under each order, every move a joined credential read as the suffix or the member report (447, 149 under each order, before the initials Except asked each chunk). The comparator is the base tree's own readings of the same grid, parsed with each tree's `nameparser.__file__` asserted. In the local corpora (`snapshot.py --local --base `, on the first build): ctgov-officials 3 entries move, `Franck Pages, Professor (MD-PHD)` and `Franck Pagès, Professor (MD-PHD)` and `Philippe PC Courtet, Professor(MD-PhD)` → suffix where nickname; fec-individuals 1, `FRANTZ (CAPT USN-RET), JOSEPH` → title 'CAPT', suffix 'USN-RET' where nickname 'CAPT USN-RET'; psc-individuals and orcid-researchers none. The initials Except only returns a joined word before a name word to the nickname master read, and none of the four stands there; each was parsed again on the final tree and reads as listed. diff --git a/docs/design/rules/S1-bracketed-credential-is-not-a-nickname.md b/docs/design/rules/S1-bracketed-credential-is-not-a-nickname.md index 91e7336e5..7cb912fce 100644 --- a/docs/design/rules/S1-bracketed-credential-is-not-a-nickname.md +++ b/docs/design/rules/S1-bracketed-credential-is-not-a-nickname.md @@ -14,8 +14,8 @@ "John (Sr.) Smith" → suffix="Sr." "John Smith (Mr, PhD)" → title="Mr" "John Smith (MD/PhD)" → suffix="MD/PhD" - "John (MD-PhD) Smith" → suffix="MD-PhD" - "John (M.D./Ph.D.) Smith" → suffix="M.D./Ph.D." + "John Smith (MD-PhD)" → suffix="MD-PhD" + "Smith (M.D./Ph.D.), John" → suffix="M.D./Ph.D." "John Smith (King)" → nickname="King" · boundary "John Smith (MD/Bob)" → nickname="MD/Bob" · boundary "John Smith (MA/PhD)" → nickname="MA/PhD" · boundary @@ -69,16 +69,19 @@ "Alex Green, FRCS (SN)" → nickname="SN" "John Smith, MD (Sr)" → suffix="MD Sr" · boundary ``` -- **A credential written as initials before a name word of its comma part** keeps its pair's reading: one or two letters with or without a period, or single letters each with its period, there are more likely the initials the person goes by, S13's reason for a member inside the name. A credential written as initials before a name word stays the nickname, and so does any word of its clause; a word that is a title too counts only written with a period after each letter, or in capitals in a name of mixed case, capitals in a name written in one case saying nothing (S6), and a generation word not at all. +- **A credential written as initials before a name word of its comma part** keeps its pair's reading: one or two letters with or without a period, or single letters each with its period, there are more likely the initials the person goes by, S13's reason for a member inside the name. A credential written as initials before a name word stays the nickname, and so does any word of its clause; a word that is a title too counts only written with a period after each letter, or in capitals in a name of mixed case, capitals in a name written in one case saying nothing (S6), and a generation word not at all; a credential joined by a slash or a hyphen counts where any of its chunks would, as its words written apart do. ``` "Justin (JP) Stearns" → nickname="JP" "Justin (R.N.) Stearns" → nickname="R.N." "Justin (MD.) Stearns" → nickname="MD." "Justin (JP, PhD) Stearns" → nickname="JP, PhD" "John (,JP) Smith" → nickname=",JP" + "Justin (JP/MD) Stearns" → nickname="JP/MD" + "John (MD-PhD) Smith" → nickname="MD-PhD" "Robert (RA) Allen" → nickname="RA" "ROBERT (RA) ALLEN" → suffix="RA" · boundary "John (Sr) Smith" → suffix="Sr" · boundary + "John (Jr/Sr) Smith" → suffix="Jr/Sr" · boundary "Smith (MD), John" → suffix="MD" · boundary ``` - **A clause holding a word the vocabulary reads as a name, or does not know**, is no such clause, and reads as its pair reads it, a nickname in a nickname pair. A member of the ambiguous class alone is S13's. diff --git a/docs/design/rules/S13-bracketed-member-settled-by-its-writing.md b/docs/design/rules/S13-bracketed-member-settled-by-its-writing.md index 560c60e96..6db56acb4 100644 --- a/docs/design/rules/S13-bracketed-member-settled-by-its-writing.md +++ b/docs/design/rules/S13-bracketed-member-settled-by-its-writing.md @@ -66,4 +66,4 @@ --- **Signal:** position: a member of the ambiguous class alone in brackets or quotes after the name; hub of S1, S3, S6 -**history:** decisions/S13 · **implemented:** nameparser/_pipeline/_post_rules.py +**history:** decisions/S13 · **implemented:** nameparser/_pipeline/_post_rules.py, nameparser/_pipeline/_vocab.py diff --git a/nameparser/_pipeline/_extract.py b/nameparser/_pipeline/_extract.py index c7902c58f..fd8ccd1dc 100644 --- a/nameparser/_pipeline/_extract.py +++ b/nameparser/_pipeline/_extract.py @@ -71,7 +71,6 @@ def _set_apart(text: str, inner: Span, kinds = [] for word in words: kind = set_apart_kind(word, lexicon) - # a member settled by its writing is weighed in post_rules if kind is None or kind == "member": return False kinds.append(kind) diff --git a/nameparser/_pipeline/_post_rules.py b/nameparser/_pipeline/_post_rules.py index dae8a423c..aa8267fdd 100644 --- a/nameparser/_pipeline/_post_rules.py +++ b/nameparser/_pipeline/_post_rules.py @@ -37,7 +37,7 @@ under every policy and reads Policy.patronymic_rules for its Turkic guard), and Lexicon.suffix_acronyms_ambiguous (a dotted member in a clause, which -carries no tag there, S13), and for S1's set-apart clause +carries no tag there, S13, through `_vocab.dotted_member`), and for S1's set-apart clause Lexicon.suffix_words directly and, through `_vocab.set_apart_kind` and `set_apart`, Lexicon.suffix_words_ambiguous, suffix_acronyms, titles, title_modifiers, title_styles and after_given_titles. @@ -52,7 +52,6 @@ from __future__ import annotations import bisect -from typing import Literal import re from collections.abc import Sequence @@ -72,9 +71,10 @@ WorkToken, _NEVER_FLIPPED, comma_bucket, copy_with, reported_suffix_fork, ) from nameparser._pipeline._vocab import ( - _DOTTED_INITIALS, delimiter_cores, has_patronymic_ending, is_initial, - is_listed_given_name, is_patronymic_marker, set_apart, set_apart_kind, suffix_reading, - unit_ends, written_as_initials, + SetApartKind, delimiter_cores, dotted_member, + has_patronymic_ending, is_initial, is_listed_given_name, + is_patronymic_marker, reads_as_initials, set_apart, set_apart_kind, + suffix_reading, unit_ends, ) from nameparser._policy import PatronymicRule from nameparser._types import ( @@ -684,7 +684,7 @@ def _set_apart_clauses(state: ParseState, tokens[i].role not in (Role.NICKNAME, Role.MAIDEN) for i in range(lo, hi)): continue - kinds: list[Literal["title", "suffix", "dual"]] = [] + kinds: list[SetApartKind] = [] member = False for i in range(lo, hi): token = tokens[i] @@ -703,17 +703,7 @@ def _set_apart_clauses(state: ParseState, # comma part stays the nickname if member and not 0 <= last_name < lo: continue - # a word that is a title too counts written with a period - # after each letter ('R.A.'), or in capitals in a name of - # mixed case ('RA'): Title case reads as the title it also - # is ('Lt'), and capitals in a name written in one case say - # nothing (S6) - if any((kind == "suffix" or kind == "dual" and ( - _DOTTED_INITIALS.fullmatch(w.strip(",")) - or state.one_case is False - and w.strip(",").isupper())) - and written_as_initials(w) - and _normalize(w.strip(",")) not in lex._sets.suffix_words + if any(reads_as_initials(w, kind, state.one_case, lex) for w, kind in zip(words, kinds)): if ahead is None: ahead = _name_ahead(state, tokens) @@ -772,18 +762,15 @@ def _report_delimited_members(tokens: list[WorkToken], lexicon: Lexicon, included. Reported here, where the clause is decided, rather than in extract, which has no token index to point at. A dotted member carries no tag in a clause (classify tags a shape only outside - one), so it is found by its text, through `set_apart_kind` as the - settling finds it.""" + one), so it is found by its text, by `dotted_member` as the + settling finds it, whatever else the vocabulary reads it as.""" for i, token in enumerate(tokens): if token.role is not Role.NICKNAME: continue - # a period inside the word ('M.A'), a comma the clause kept on - # it left off; one ending it is extract's escape. The period is - # asked first, at C level, so a nickname written without one - # pays no frame + # the period asked first, at C level, so a nickname written + # without one pays no frame if (AMBIGUOUS_ACRONYM_TAG in token.tags - or "." in token.text - and set_apart_kind(token.text, lexicon) == "member"): + or "." in token.text and dotted_member(token.text, lexicon)): ambiguities.append(PendingAmbiguity( AmbiguityKind.SUFFIX_OR_NICKNAME, f"delimited {token.text!r} is also a post-nominal; read " diff --git a/nameparser/_pipeline/_vocab.py b/nameparser/_pipeline/_vocab.py index 6a83d9e52..7741df332 100644 --- a/nameparser/_pipeline/_vocab.py +++ b/nameparser/_pipeline/_vocab.py @@ -398,29 +398,53 @@ def ends_in_a_generation(texts: Sequence[str], one_case: bool | None, and reads_as_generation(last, frozenset(), texts[-2], one_case)) +#: What a word of a clause S1 sets apart is: a title, a suffix, or a +#: word that is both, its place in the name choosing (rules/S1). +SetApartKind = Literal["title", "suffix", "dual"] + + +# rules/S13: "Written with a period inside it, whatever case the name is +# in" -- the writing that settles a member, asked of its text since a +# clause word carries no shape tag (decisions/S13, #810, #1088) +def dotted_member(word: str, lexicon: Lexicon) -> bool: + """Whether a word of a delimited clause is a member of the + ambiguous class (S4) written with a period inside it ('M.A', + 'M.A.'), a comma the clause keeps on it left off (rules/N1). A + period ending the clause is extract's escape, not this.""" + bare = word.strip(",") + return ("." in bare.strip(".") + and _normalize(bare).replace(".", "") + in lexicon._sets.suffix_acronyms_ambiguous) + + # rules/S1: "a clause wholly of words the vocabulary reads as no name # is set apart from the name" -- the word's half; `set_apart` is the # clause's. Its suffix half is `suffix_as_written`'s, so a credential -# joined by a slash or a hyphen is one here as it is bare, a member of -# the ambiguous class asked first and left to S13's position test -# (decisions/S1, #792, #1088) +# joined by a slash or a hyphen is one here as it is bare, a dotted +# member asked first and left to S13's position test (decisions/S1, +# #792, #1088) def set_apart_kind(word: str, lexicon: Lexicon, - ) -> Literal["title", "suffix", "dual", "member"] | None: + ) -> SetApartKind | Literal["member"] | None: """What a word in a delimited clause is to S1's set-apart reading: a title, a suffix (a word `suffix_as_written` reads, joined credentials included), both, a member of the ambiguous class - written with a period inside it, which S13 settles after the name - only, or None where the vocabulary reads it as a name or not at - all. A comma the clause keeps on its word is no part of it - (rules/N1).""" + written with a period inside it (`dotted_member`), which S13 + settles after the name only, or None where the vocabulary reads it + as a name or not at all. A comma the clause keeps on its word is no + part of it (rules/N1).""" bare = word.strip(",") folded = _normalize(bare) sets = lexicon._sets title = folded in sets.titles + # a member of the ambiguous class, the set asked inline as the + # necessary half of `dotted_member`, so no other word pays its + # frame: `suffix_as_written` would read a dotted member as a suffix + # wherever it stands, and reads no other member as one if folded.replace(".", "") in sets.suffix_acronyms_ambiguous: if title: return "title" - return "member" if "." in bare.strip(".") else None + return "member" if "." in bare and dotted_member( + bare, lexicon) else None suffix = suffix_as_written(folded, bare, lexicon) if title: return "dual" if suffix else "title" @@ -438,16 +462,39 @@ def set_apart_kind(word: str, lexicon: Lexicon, # rules/S1: "A credential written as initials before a name word stays -# the nickname" -- the brackets more likely hold the initials the person -# goes by there; decisions/S1 (#792) -def written_as_initials(word: str) -> bool: - """Whether a word of a delimited clause is written in the shape of - a person's initials, a comma it keeps left off.""" - return _INITIALS_SHAPE.fullmatch(word.strip(",")) is not None +# the nickname, and so does any word of its clause" -- asked of each +# chunk of a credential joined by a slash or a hyphen, so it reads +# before a name word as its split spelling does (decisions/S1, #792, +# #1088) +def reads_as_initials(word: str, kind: SetApartKind, + one_case: bool | None, lexicon: Lexicon) -> bool: + """Whether a word of a clause S1 would set apart counts, before a + name word of its comma part, as the initials the person goes by: a + credential written in their shape that is no generation word, and a + word that is a title too only written with a period after each + letter ('R.A.') or in capitals in a name of mixed case ('RA'), + Title case reading as the title it also is and capitals in a name + written in one case saying nothing (S6). `one_case` is + ParseState.one_case. A joined credential counts where any of its + chunks would ('JP/MD', 'MD/PhD'), as its words written apart do.""" + bare = word.strip(",") + if kind == "suffix" and ("/" in bare or "-" in bare): + for chunk in _JOINERS.split(bare): + ck = set_apart_kind(chunk, lexicon) + if ck is not None and ck != "title" and reads_as_initials( + chunk, "suffix" if ck == "member" else ck, one_case, + lexicon): + return True + return False + return ((kind == "suffix" or kind == "dual" and ( + _DOTTED_INITIALS.fullmatch(bare) is not None + or one_case is False and bare.isupper())) + and _INITIALS_SHAPE.fullmatch(bare) is not None + and _normalize(bare) not in lexicon._sets.suffix_words) def set_apart(words: Sequence[str], - kinds: Sequence[Literal["title", "suffix", "dual"]], + kinds: Sequence[SetApartKind], behind_a_credential: bool, lexicon: Lexicon) -> bool: """rules/S1's clause tests over the words of a delimited clause, each already found to be no name by `set_apart_kind` (or, after diff --git a/tests/v2/cases.py b/tests/v2/cases.py index f775ed726..dcf0b4295 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -9805,25 +9805,39 @@ def _check_cjk_shape_purity(self) -> None: classification="fix(#1088)", notes="bare 'John Smith MD/PhD' reads suffix (rules/S3); 1.4.0 " "through 2.8 read nickname 'MD/PhD'"), - Case("paren_hyphen_joined_credential_inside_the_name_set_apart", + Case("paren_hyphen_joined_credential_inside_the_name_stays_the_nickname", "John (MD-PhD) Smith", - {"given": "John", "family": "Smith", "suffix": "MD-PhD"}, + {"given": "John", "family": "Smith", "nickname": "MD-PhD"}, + notes="S1's initials Except asks each chunk, and MD is written as " + "initials, so it reads as 'John (MD, PhD) Smith' does " + "(#1088)"), + Case("paren_joined_initials_and_credential_stays_the_nickname", + "Justin (JP/MD) Stearns", + {"given": "Justin", "family": "Stearns", "nickname": "JP/MD"}, + notes="the initials the person goes by, as 'Justin (JP, MD) " + "Stearns' reads (#1088)"), + Case("paren_joined_generations_inside_the_name_set_apart", + "John (Jr/Sr) Smith", + {"given": "John", "family": "Smith", "suffix": "Jr/Sr"}, classification="fix(#1088)", - notes="no initials' shape, so S1's initials Except does not " - "reach it; 1.4.0 through 2.8 read nickname 'MD-PhD'"), + notes="no chunk is written as initials, a generation word " + "counting not at all, so it reads as 'John (Jr, Sr) Smith' " + "does; 1.4.0 through 2.8 read nickname 'Jr/Sr'"), Case("paren_joined_credential_before_a_family_comma_set_apart", "Smith (MD/PhD), John", {"given": "John", "family": "Smith", "suffix": "MD/PhD"}, classification="fix(#1088)", notes="as 'Smith (MD), John' reads; 1.4.0 through 2.8 read " "nickname 'MD/PhD'"), - Case("paren_dotted_joined_credential_set_apart_not_escaped", + Case("paren_dotted_joined_credential_inside_the_name_stays_the_nickname", "John (M.D./Ph.D.) Smith", - {"given": "John", "family": "Smith", "suffix": "M.D./Ph.D."}, + {"given": "John", "family": "Smith", "nickname": "M.D./Ph.D."}, classification="fix(#1088)", - notes="set apart before the period escape is asked, which had " - "read it bare: 1.4.0 through 2.8 read middle " - "'M.D./Ph.D.'"), + notes="M.D. is written as initials, so the initials Except keeps " + "the nickname, as 'John (M.D., Ph.D.) Smith' reads; a clause " + "it declines keeps its pair's reading, not the period " + "escape, which had read it bare: 1.4.0 through 2.8 read " + "middle 'M.D./Ph.D.'"), Case("paren_dotted_joined_credential_after_a_lone_name_word", "Jack (M.D./Ph.D.)", {"family": "Jack", "suffix": "M.D./Ph.D."}, @@ -9831,6 +9845,82 @@ def _check_cjk_shape_purity(self) -> None: notes="the lone word is N3's family name, as in 'Jack (PhD)'; " "2.8 read given 'Jack', reporting given-or-family, and " "1.4.0 last 'M.D./Ph.D.'"), + Case("paren_joined_credential_before_a_suffix_comma", + 'John Smith (MD/PhD), Jr.', + {'given': 'John', 'family': 'Smith', 'suffix': 'MD/PhD, Jr.'}, + classification="fix(#1088)", + notes="the comma parts the clause's suffix from the one after it; 1.4.0 through 2.8 read nickname 'MD/PhD'"), + Case("paren_joined_credential_after_a_suffix_comma", + 'John Smith, Jr. (MD/PhD)', + {'given': 'John', 'family': 'Smith', 'suffix': 'Jr. MD/PhD'}, + classification="fix(#1088)", + notes="1.4.0 through 2.8 read nickname 'MD/PhD'"), + Case("paren_joined_credential_after_a_family_comma", + 'Smith, John (MD-PhD)', + {'given': 'John', 'family': 'Smith', 'suffix': 'MD-PhD'}, + classification="fix(#1088)", + notes="1.4.0 through 2.8 read nickname 'MD-PhD'"), + Case("paren_joined_credential_after_an_empty_head", + ', John (MD/PhD)', + {'given': 'John', 'suffix': 'MD/PhD'}, + classification="fix(#1088)", + notes="1.4.0 through 2.8 read nickname 'MD/PhD'"), + Case("paren_joined_credential_alone_before_a_comma", + '(MD/PhD), John', + {'given': 'John', 'suffix': 'MD/PhD'}, + classification="fix(#1088)", + notes="its comma part holds no name word, so S1's initials Except is not asked; 1.4.0 through 2.8 read nickname 'MD/PhD'"), + Case("paren_joined_credential_alone", + '(MD/PhD)', + {'suffix': 'MD/PhD'}, + classification="fix(#1088)", + notes="a name of no word of its own is left empty, as '(MPH)' is (S1's Accepted); 1.4.0 through 2.8 read nickname 'MD/PhD'"), + Case("quoted_joined_credential_set_apart", + 'John Smith "MD/PhD"', + {'given': 'John', 'family': 'Smith', 'suffix': 'MD/PhD'}, + classification="fix(#1088)", + notes="quotes set it apart as brackets do; 1.4.0 through 2.8 read nickname 'MD/PhD'"), + Case("paren_joined_credential_family_first", + 'Smith John (MD/PhD)', + {'given': 'John', 'family': 'Smith', 'suffix': 'MD/PhD'}, + policy=Policy(name_order=FAMILY_FIRST), + classification="fix(#1088)", + notes="the clause reads alike under every order; 1.4.0 through 2.8 read nickname 'MD/PhD'"), + Case("paren_joined_credential_beside_a_member_settles_the_clause", + 'John Smith (MD/PhD, MBA)', + {'given': 'John', 'family': 'Smith', 'suffix': 'MD/PhD, MBA'}, + classification="fix(#1088)", + notes="MBA is settled by its capitals (S13), so the clause is set apart and reports nothing: 2.8 read nickname 'MD/PhD, MBA', reporting suffix-or-nickname, and 1.4.0 the nickname"), + Case("paren_joined_credential_after_a_listed_one", + 'John Smith (PhD, MD/MPH)', + {'given': 'John', 'family': 'Smith', 'suffix': 'PhD, MD/MPH'}, + classification="fix(#1088)", + notes="1.4.0 through 2.8 read nickname 'PhD, MD/MPH'"), + Case("paren_joined_credential_keeping_a_comma", + 'John Smith (MD/PhD,)', + {'given': 'John', 'family': 'Smith', 'suffix': 'MD/PhD'}, + classification="fix(#1088)", + notes="the comma the clause keeps is no part of the word (rules/N1); 1.4.0 through 2.8 read nickname 'MD/PhD,'"), + Case("paren_dotted_member_keeping_a_comma_before_a_family_comma", + 'Smith (M.A,), Jean', + {'given': 'Jean', 'family': 'Smith', 'nickname': 'M.A,'}, + classification="fix(#1088)", + ambiguities=('suffix-or-nickname',), + notes="reported as 'Smith (MA), John' is; 2.8 reported nothing, and 1.4.0 read nickname 'M.A,'"), + Case("paren_dotted_member_keeping_a_comma_after_its_period", + 'Jean (M.A.,) Smith', + {'given': 'Jean', 'family': 'Smith', 'nickname': 'M.A.,'}, + classification="fix(#1088)", + ambiguities=('suffix-or-nickname',), + notes="the comma keeps the period off the clause's end, so no escape; reported as 'Jean (M.A) Smith' is; 2.8 reported nothing, and 1.4.0 read nickname 'M.A.,'"), + Case("paren_period_final_member_inside_the_name_escapes", + 'Jean (M.A.) Smith', + {'given': 'Jean', 'middle': 'M.A.', 'family': 'Smith'}, + notes="a dotted member is no word extract sets apart: S13 weighs it once the name's end is known, and here the period escape reads it bare"), + Case("paren_period_final_member_before_a_family_comma_escapes", + 'Smith (M.A.), John', + {'given': 'John', 'family': 'Smith', 'suffix': 'M.A.'}, + notes="as the row above: the period escape, not S1's set-apart reading, which would make it a reported nickname before the comma"), Case("paren_joined_credential_and_name_stays_the_nickname", "John Smith (MD/Bob)", {"given": "John", "family": "Smith", "nickname": "MD/Bob"}, diff --git a/tests/v2/pipeline/test_post_rules.py b/tests/v2/pipeline/test_post_rules.py index f7602864e..ff2786ab5 100644 --- a/tests/v2/pipeline/test_post_rules.py +++ b/tests/v2/pipeline/test_post_rules.py @@ -1772,8 +1772,9 @@ def test_a_set_apart_clause_trims_its_comma_off_the_word(text: str) -> None: #: rules/S1's set-apart word asks `suffix_as_written`, the question a #: bare word is asked, so a credential S3 reads joined by a slash or a -#: hyphen is set apart in brackets as its split spelling is (#1088): -#: each word bracketed after the name and inside it. +#: hyphen is set apart in brackets after the name as its split spelling +#: is (#1088). Inside the name the initials Except decides, which the +#: two-spelling invariant below reads. _BRACKETED_WORDS = ("PhD", "Jr.", "MD/PhD", "MD-PhD", "md/phd", "Md-PhD", "MD/PhD/MPH", "M.D./Ph.D.", "M.A./PhD", "Jr/Sr", "MD/Bob", "MA/PhD", "Smith-Jones", "Ae-Ra", "I-V", "MD/") @@ -1781,44 +1782,124 @@ def test_a_set_apart_clause_trims_its_comma_off_the_word(text: str) -> None: #: The rows the guard moves, as the tree before #1088 read them, the #: set-apart word asking its own copy of the unambiguous suffix sets: #: the test's negative control, recorded 2026-11-05 from a detached -#: worktree of 4109b026 as (suffix, nickname). The joined words were -#: nicknames, and the dotted one inside the name took the period escape -#: and read as the middle name. +#: worktree of 4109b026 as (suffix, nickname). _UNGUARDED_EFFECT_1088 = { - "John Smith (MD/PhD)": ("", "MD/PhD"), - "John (MD/PhD) Smith": ("", "MD/PhD"), - "John Smith (MD-PhD)": ("", "MD-PhD"), - "John (MD-PhD) Smith": ("", "MD-PhD"), - "John Smith (md/phd)": ("", "md/phd"), - "John (md/phd) Smith": ("", "md/phd"), - "John Smith (Md-PhD)": ("", "Md-PhD"), - "John (Md-PhD) Smith": ("", "Md-PhD"), - "John Smith (MD/PhD/MPH)": ("", "MD/PhD/MPH"), - "John (MD/PhD/MPH) Smith": ("", "MD/PhD/MPH"), - "John (M.D./Ph.D.) Smith": ("", ""), - "John Smith (M.A./PhD)": ("", "M.A./PhD"), - "John (M.A./PhD) Smith": ("", "M.A./PhD"), - "John Smith (Jr/Sr)": ("", "Jr/Sr"), - "John (Jr/Sr) Smith": ("", "Jr/Sr"), + "MD/PhD": ("", "MD/PhD"), + "MD-PhD": ("", "MD-PhD"), + "md/phd": ("", "md/phd"), + "Md-PhD": ("", "Md-PhD"), + "MD/PhD/MPH": ("", "MD/PhD/MPH"), + "M.A./PhD": ("", "M.A./PhD"), + "Jr/Sr": ("", "Jr/Sr"), } +#: The grid's words S3 refuses, pinned apart from the predicate the +#: test asks, so a refusal S3 lost would not move both sides at once. +_REFUSED_JOINED = frozenset({"MD/Bob", "MA/PhD", "Smith-Jones", "Ae-Ra", + "I-V", "MD/"}) + @pytest.mark.parametrize("word", _BRACKETED_WORDS) def test_a_bracketed_word_is_the_suffix_where_the_bare_word_is_one( word: str) -> None: lexicon = Lexicon.default() suffix = suffix_as_written(_normalize(word), word, lexicon) - for text in (f"John Smith ({word})", f"John ({word}) Smith"): - name = parse(text) - read = (name.suffix, name.nickname) - assert read == ((word, "") if suffix else ("", word)), text - if text in _UNGUARDED_EFFECT_1088: - assert _UNGUARDED_EFFECT_1088[text] != read + assert suffix is (word not in _REFUSED_JOINED) + name = parse(f"John Smith ({word})") + read = (name.suffix, name.nickname) + assert read == ((word, "") if suffix else ("", word)) + if word in _UNGUARDED_EFFECT_1088: + assert _UNGUARDED_EFFECT_1088[word] != read -def test_the_bracketed_word_control_names_rows_of_the_grid() -> None: +def test_the_bracketed_word_control_names_words_of_the_grid() -> None: # a control row outside the grid would guard nothing - texts = {t for w in _BRACKETED_WORDS - for t in (f"John Smith ({w})", f"John ({w}) Smith")} - assert set(_UNGUARDED_EFFECT_1088) <= texts - assert len(_UNGUARDED_EFFECT_1088) == 15 + assert set(_UNGUARDED_EFFECT_1088) <= set(_BRACKETED_WORDS) + assert len(_UNGUARDED_EFFECT_1088) == 7 + + +def test_a_dotted_member_a_caller_lists_as_a_title_still_reports() -> None: + # rules/S13's report asks the member's writing whatever else the + # vocabulary reads it as: a caller listing 'm.a' as a title makes a + # title-only clause behind a credential, which keeps its pair's + # reading (rules/S1), and the member left the nickname still + # reports (#1088's review: the first draft read the report off the + # set-apart kind, which answers "title" there) + parser = Parser(lexicon=Lexicon.default().add("m.a", kind="title")) + for text in ("John Smith PhD (M.A)", "John Smith PhD (M.A.)"): + name = parser.parse(text) + assert name.nickname == text[16:-1] + assert [a.kind for a in name.ambiguities] == [ + AmbiguityKind.SUFFIX_OR_NICKNAME], text + + +def test_a_member_a_caller_lists_as_a_title_is_the_title() -> None: + # a title outranks a member in a set-apart clause, as it did before + # #1088: unreachable with the default vocabulary, which lists no + # member of the ambiguous class as a title + parser = Parser(lexicon=Lexicon.default().add("aca", kind="title")) + assert "aca" in Lexicon.default()._sets.suffix_acronyms_ambiguous + name = parser.parse("John Smith (Aca)") + assert (name.title, name.nickname) == ("Aca", "") + + +#: rules/S1's two spellings of a credential pair, joined by a slash or a +#: hyphen and written apart with a comma, read alike in a clause, the +#: initials Except asking each chunk of the joined word (#1088, axis 11 +#: of docs/design/AGENTS.md: an invariant over two inputs). The chunks +#: are credentials S1 sets apart with no other reading -- a word that +#: is a title too, or a member of the ambiguous class, reads by its own +#: rules written apart -- and the second never ends in a period, which +#: would hand the comma spelling the period escape. +_PAIR_CHUNKS = ("MD", "PhD", "JP", "Jr", "M.D.", "RN", "Esq") +_PAIR_SHAPES = ("John Smith ({})", "John ({}) Smith", "({}) John Smith", + "({})", "Smith ({}), John", "Smith, John ({})", + "JOHN ({}) SMITH", "Jack ({})", "John Smith ({}), Jr.", + ", John ({})", "John Smith, Jr. ({})") +_PAIR_PARSERS = (Parser(), Parser(policy=Policy(name_order=FAMILY_FIRST)), + Parser(policy=Policy(name_order=FAMILY_FIRST_GIVEN_LAST))) + +#: How many of those rows read apart with the initials Except asking +#: the whole joined word, as the first draft of #1088 did: the negative +#: control, recorded 2026-11-05 with `reads_as_initials`' chunk loop +#: switched off. Each is a joined pair inside or before a name word of +#: its comma part, set apart where its spelling written apart stays the +#: nickname. +_UNGUARDED_PAIRS_1088 = 540 + + +#: How many (pair, shape, order) rows the invariant reads, so a grid +#: trimmed to nothing cannot pass it. +_PAIRS_CHECKED = 2310 + + +def _pair_differences() -> list[str]: + found = [] + checked = 0 + for a in _PAIR_CHUNKS: + for b in _PAIR_CHUNKS: + if a == b or b.endswith("."): + continue + for joined in (f"{a}/{b}", f"{a}-{b}"): + # a pair S3 reads as no suffix bare ('Jr-Esq', written as + # a hyphenated name is) is no joined credential at all + if not suffix_as_written(_normalize(joined), joined, + Lexicon.default()): + continue + for shape in _PAIR_SHAPES: + for parser in _PAIR_PARSERS: + apart = f"{a}, {b}" + one = {k: v.replace(joined, apart) for k, v in + parser.parse(shape.format(joined)) + .as_dict().items()} + if one != parser.parse(shape.format(apart)).as_dict(): + found.append(shape.format(joined)) + checked += 1 + assert checked == _PAIRS_CHECKED + return found + + +def test_a_joined_credential_reads_as_its_spelling_written_apart() -> None: + assert _pair_differences() == [] + # the control is a share of the grid the guard reads + assert 0 < _UNGUARDED_PAIRS_1088 < _PAIRS_CHECKED diff --git a/tests/v2/rules_budget.json b/tests/v2/rules_budget.json index a0c549f2b..8cc33fc9e 100644 --- a/tests/v2/rules_budget.json +++ b/tests/v2/rules_budget.json @@ -64,7 +64,7 @@ "R9": {"words": 397, "then_words": 122, "because_sentences": 3, "except": 5, "precedence": 0, "accepted": 13, "accepted_words": 477}, "R10": {"words": 168, "then_words": 26, "because_sentences": 2, "except": 2, "precedence": 0, "accepted": 6, "accepted_words": 160}, "R11": {"words": 226, "then_words": 42, "because_sentences": 3, "except": 2, "precedence": 0, "accepted": 5, "accepted_words": 195}, - "S1": {"words": 714, "then_words": 248, "because_sentences": 3, "except": 6, "precedence": 3, "accepted": 5, "accepted_words": 173}, + "S1": {"words": 736, "then_words": 248, "because_sentences": 3, "except": 6, "precedence": 3, "accepted": 5, "accepted_words": 173}, "S2": {"words": 658, "then_words": 120, "because_sentences": 1, "except": 0, "precedence": 7, "accepted": 3, "accepted_words": 222}, "S3": {"words": 960, "then_words": 398, "because_sentences": 3, "except": 4, "precedence": 5, "accepted": 6, "accepted_words": 357}, "S4": {"words": 306, "then_words": 164, "because_sentences": 2, "except": 0, "precedence": 3, "accepted": 0, "accepted_words": 0}, diff --git a/tests/v2/test_cases.py b/tests/v2/test_cases.py index 25babe8a3..f43ff0afc 100644 --- a/tests/v2/test_cases.py +++ b/tests/v2/test_cases.py @@ -364,9 +364,10 @@ def test_the_field_sweep_sees_the_claims_it_checks() -> None: # 2026-11-05, #1080's two boundary rows and its paired-initials row, # each reporting suffix-or-name under three orders; 1606 -> 1609 on # 2026-11-05, #1088's 'Jean (M.A,) Smith' row reporting - # suffix-or-nickname under three orders. - assert claims == 1609, ( - f"the field sweep checks {claims} claims, recorded as 1609 on " + # suffix-or-nickname under three orders; 1609 -> 1615 the same day, + # its review's two kept-comma member rows, three each. + assert claims == 1615, ( + f"the field sweep checks {claims} claims, recorded as 1615 on " f"2026-11-05") diff --git a/tests/v2/test_layering.py b/tests/v2/test_layering.py index ac586a36b..e0b1caa61 100644 --- a/tests/v2/test_layering.py +++ b/tests/v2/test_layering.py @@ -651,7 +651,9 @@ def test_the_check_finds_a_tool_read_through_an_alias( #: taking extract's four suffix-set loads out, 119 -> 128; #1088's #: set-apart word asking `suffix_as_written`, three loads out of #: `_vocab` and both of `_post_rules`' copies of the dotted member -#: test, 128 -> 123); the tools' 19 are 17 +#: test, 128 -> 123; its parity fix moving the initials test into +#: `_vocab.reads_as_initials` and the dotted test into +#: `_vocab.dotted_member`, 123 -> 124); the tools' 19 are 17 #: attribute loads, every tool scanned, and config_diff.py's #: `sets =` binding and the roster `getattr` through it (its #: `changed_words`, which compares two lexicons set by set), which @@ -661,7 +663,7 @@ def test_the_check_finds_a_tool_read_through_an_alias( #: `_lexicon._fields_of` from 4i-b until 4i-c retired it. #: A reader added through `_sets` moves these, and they are re-taken #: then: they count what the check would refuse with the move undone. -_UNGUARDED_READS = {"library": 123, "tools": 19} +_UNGUARDED_READS = {"library": 124, "tools": 19} def test_the_check_refuses_every_read_4i_a_moved() -> None: diff --git a/tests/v2/test_regex_sync.py b/tests/v2/test_regex_sync.py index 72ccc5e9c..c2e810c83 100644 --- a/tests/v2/test_regex_sync.py +++ b/tests/v2/test_regex_sync.py @@ -71,10 +71,10 @@ def test_dotted_initial_is_the_period_alternative_of_initial() -> None: ("_vocab", "_PERIOD_ABBREV"): None, # rules/S1's initials shape for a bracketed credential (#792) ("_vocab", "_INITIALS_SHAPE"): None, - # its dotted alternative, from which _INITIALS_SHAPE is built; the - # SAME object in _post_rules, which imports it for S1's dual words + # its dotted alternative, from which _INITIALS_SHAPE is built, read + # for S1's dual words by `reads_as_initials` beside it (#1088 moved + # the test there from _post_rules, which imported it) ("_vocab", "_DOTTED_INITIALS"): None, - ("_post_rules", "_DOTTED_INITIALS"): ("_vocab", "_DOTTED_INITIALS"), # SAME object as _vocab's, not a second copy: _pieces imports it # from _vocab (#289/#516, quality-review finding -- is_title_shaped # moved there so name_word_count could share is_leading_title's H2 diff --git a/tools/differential/corpus_rules.jsonl b/tools/differential/corpus_rules.jsonl index 6fbbd90d1..970f1091b 100644 --- a/tools/differential/corpus_rules.jsonl +++ b/tools/differential/corpus_rules.jsonl @@ -469,6 +469,7 @@ "Jane „JD Smith" "Jean 'JD' Smith" "Jean (M.A) Smith" +"Jean (M.A,) Smith" "Jean DUPONT" "Jean Marie Fils" "Jean Pierre DUPONT" @@ -480,6 +481,8 @@ "John (,JP) Smith" "John (Dr.) Smith" "John (Jack) Smith" +"John (Jr/Sr) Smith" +"John (MD-PhD) Smith" "John (Sr) Smith" "John (Sr.) Smith" "John . Smith" @@ -518,9 +521,13 @@ "John Smith (MA) Ph. D." "John Smith (MA), PhD" "John Smith (MA)." +"John Smith (MA/PhD)" "John Smith (MBA, PhD)" "John Smith (MD ,PhD)" "John Smith (MD,) (PhD)" +"John Smith (MD-PhD)" +"John Smith (MD/Bob)" +"John Smith (MD/PhD)" "John Smith (Ma)" "John Smith (Mr)" "John Smith (Mr, PhD)" @@ -796,6 +803,7 @@ "Juan Ó. Pérez" "Justin (JP) Stearns" "Justin (JP, PhD) Stearns" +"Justin (JP/MD) Stearns" "Justin (MD.) Stearns" "Justin (R.N.) Stearns" "KIM, AE-RA" @@ -1069,6 +1077,7 @@ "Siti binti Ahmad binti Hasan" "Smith" "Smith (Jones)" +"Smith (M.D./Ph.D.), John" "Smith (MA), John" "Smith (MD), John" "Smith (née Jones)" diff --git a/tools/differential/snapshot.jsonl b/tools/differential/snapshot.jsonl index 6d405a572..7facf832d 100644 --- a/tools/differential/snapshot.jsonl +++ b/tools/differential/snapshot.jsonl @@ -1684,6 +1684,7 @@ {"name": "Javier López Díaz", "given": "Javier", "middle": "López", "family": "Díaz", "initials": "J. L. D."} {"name": "Jean 'JD' Smith", "given": "Jean", "family": "Smith", "nickname": "JD", "ambiguities": ["SUFFIX_OR_NICKNAME"], "alternatives": ["SUFFIX_OR_NICKNAME: credential as suffix (vocabulary)"], "initials": "J. S.", "capitalized_force": "Jean \"Jd\" Smith"} {"name": "Jean (M.A) Smith", "given": "Jean", "family": "Smith", "nickname": "M.A", "ambiguities": ["SUFFIX_OR_NICKNAME"], "alternatives": ["SUFFIX_OR_NICKNAME: credential as suffix (vocabulary)"], "initials": "J. S."} +{"name": "Jean (M.A,) Smith", "given": "Jean", "family": "Smith", "nickname": "M.A,", "ambiguities": ["SUFFIX_OR_NICKNAME"], "alternatives": ["SUFFIX_OR_NICKNAME: credential as suffix (vocabulary)"], "initials": "J. S."} {"name": "Jean DUPONT", "given": "Jean", "family": "DUPONT", "initials": "J. D.", "capitalized_force": "Jean Dupont"} {"name": "Jean Joseph Antoine Marie de Witte", "given": "Jean", "middle": "Joseph Antoine Marie", "family": "de Witte", "initials": "J. J. A. M. W."} {"name": "Jean Louis Charles d'Orléans-Longueville", "given": "Jean", "middle": "Louis Charles", "family": "d'Orléans-Longueville", "initials": "J. L. C. d.", "capitalized_force": "Jean Louis Charles D'Orléans-Longueville"} @@ -1751,6 +1752,8 @@ {"name": "John (,JP) Smith", "given": "John", "family": "Smith", "nickname": ",JP", "initials": "J. S.", "capitalized_force": "John \",Jp\" Smith"} {"name": "John (Dr.) Smith", "title": "Dr.", "given": "John", "family": "Smith", "initials": "J. S."} {"name": "John (Jack) Smith", "given": "John", "family": "Smith", "nickname": "Jack", "initials": "J. S."} +{"name": "John (Jr/Sr) Smith", "given": "John", "family": "Smith", "suffix": "Jr/Sr", "initials": "J. S."} +{"name": "John (MD-PhD) Smith", "given": "John", "family": "Smith", "nickname": "MD-PhD", "initials": "J. S.", "capitalized_force": "John \"Md-PhD\" Smith"} {"name": "John (Sr) Smith", "given": "John", "family": "Smith", "suffix": "Sr", "initials": "J. S.", "generation": "Sr"} {"name": "John (Sr.) Smith", "given": "John", "family": "Smith", "suffix": "Sr.", "initials": "J. S.", "generation": "Sr."} {"name": "John . Smith", "given": "John", "middle": ".", "family": "Smith", "initials": "J. .. S."} @@ -1830,9 +1833,13 @@ {"name": "John Smith (MA) Ph. D.", "given": "John", "family": "Smith", "suffix": "MA Ph. D.", "initials": "J. S."} {"name": "John Smith (MA), PhD", "given": "John", "family": "Smith", "suffix": "MA, PhD", "initials": "J. S."} {"name": "John Smith (MA).", "given": "John", "middle": "Smith", "family": ".", "nickname": "MA", "ambiguities": ["SUFFIX_OR_NICKNAME"], "alternatives": ["SUFFIX_OR_NICKNAME: credential as suffix (vocabulary)"], "initials": "J. S. ..", "capitalized_force": "John \"Ma\" Smith ."} +{"name": "John Smith (MA/PhD)", "given": "John", "family": "Smith", "nickname": "MA/PhD", "initials": "J. S.", "capitalized_force": "John \"Ma/PhD\" Smith"} {"name": "John Smith (MBA, PhD)", "given": "John", "family": "Smith", "suffix": "MBA, PhD", "initials": "J. S."} {"name": "John Smith (MD ,PhD)", "given": "John", "family": "Smith", "suffix": "MD, PhD", "initials": "J. S."} {"name": "John Smith (MD,) (PhD)", "given": "John", "family": "Smith", "suffix": "MD, PhD", "initials": "J. S."} +{"name": "John Smith (MD-PhD)", "given": "John", "family": "Smith", "suffix": "MD-PhD", "initials": "J. S."} +{"name": "John Smith (MD/Bob)", "given": "John", "family": "Smith", "nickname": "MD/Bob", "initials": "J. S.", "capitalized_force": "John \"Md/Bob\" Smith"} +{"name": "John Smith (MD/PhD)", "given": "John", "family": "Smith", "suffix": "MD/PhD", "initials": "J. S."} {"name": "John Smith (Ma)", "given": "John", "family": "Smith", "nickname": "Ma", "ambiguities": ["SUFFIX_OR_NICKNAME"], "alternatives": ["SUFFIX_OR_NICKNAME: credential as suffix (vocabulary)"], "initials": "J. S."} {"name": "John Smith (Mr)", "given": "John", "family": "Smith", "nickname": "Mr", "initials": "J. S."} {"name": "John Smith (Mr, PhD)", "title": "Mr", "given": "John", "family": "Smith", "suffix": "PhD", "initials": "J. S."} @@ -2292,6 +2299,7 @@ {"name": "Juro", "order": "FAMILY_FIRST", "family": "Juro", "ambiguities": ["GIVEN_OR_FAMILY"], "alternatives": ["GIVEN_OR_FAMILY: name as given (position)"], "initials": "J."} {"name": "Justin (JP) Stearns", "given": "Justin", "family": "Stearns", "nickname": "JP", "initials": "J. S.", "capitalized_force": "Justin \"Jp\" Stearns"} {"name": "Justin (JP, PhD) Stearns", "given": "Justin", "family": "Stearns", "nickname": "JP, PhD", "initials": "J. S.", "capitalized_force": "Justin \"Jp, PhD\" Stearns"} +{"name": "Justin (JP/MD) Stearns", "given": "Justin", "family": "Stearns", "nickname": "JP/MD", "initials": "J. S.", "capitalized_force": "Justin \"Jp/Md\" Stearns"} {"name": "Justin (MD.) Stearns", "given": "Justin", "family": "Stearns", "nickname": "MD.", "initials": "J. S.", "capitalized_force": "Justin \"Md.\" Stearns"} {"name": "Justin (R.N.) Stearns", "given": "Justin", "family": "Stearns", "nickname": "R.N.", "initials": "J. S."} {"name": "Justo Novo Verdes", "given": "Justo", "middle": "Novo", "family": "Verdes", "initials": "J. N. V."} @@ -3505,6 +3513,7 @@ {"name": "Slovák, Ján Matej", "given": "Ján", "middle": "Matej", "family": "Slovák", "initials": "J. M. S."} {"name": "Smith", "given": "Smith", "ambiguities": ["GIVEN_OR_FAMILY"], "alternatives": ["GIVEN_OR_FAMILY: name as family (position)"], "initials": "S."} {"name": "Smith (Jones)", "family": "Smith", "nickname": "Jones", "initials": "S."} +{"name": "Smith (M.D./Ph.D.), John", "given": "John", "family": "Smith", "suffix": "M.D./Ph.D.", "initials": "J. S."} {"name": "Smith (MA), John", "given": "John", "family": "Smith", "nickname": "MA", "ambiguities": ["SUFFIX_OR_NICKNAME"], "alternatives": ["SUFFIX_OR_NICKNAME: credential as suffix (vocabulary)"], "initials": "J. S.", "capitalized_force": "John \"Ma\" Smith"} {"name": "Smith (MD), John", "given": "John", "family": "Smith", "suffix": "MD", "initials": "J. S."} {"name": "Smith (née Jones)", "family": "Smith", "maiden": "Jones", "initials": "S."} From a686b5520bf0659f29e746525ffdb882b6c9a452 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Thu, 5 Nov 2026 16:28:28 -0800 Subject: [PATCH 3/4] Simplify #1088's set-apart word and initials test From the /simplify pass: `set_apart_kind` asks the suffix sets and the joiners inline ahead of `suffix_as_written`, the predicate's necessary half, so a nickname word pays no frame; the initials test is skipped for a clause after the last name word (a name word after it stands behind that word) and for a title word; `reads_as_initials`' one-level recursion is flat, `_shaped_as_initials` taking a member as the credential it is settled as; post_rules folds a member straight to the suffix. No reading moves (the 1,260-name grid and the snapshot are identical); `John Smith (Andy)` costs 214 frames as on master, the reference row 306.80. Co-Authored-By: Claude Opus 5.5 --- nameparser/_pipeline/_post_rules.py | 18 +++++++++------ nameparser/_pipeline/_vocab.py | 35 ++++++++++++++++++++--------- tests/v2/test_layering.py | 6 +++-- 3 files changed, 40 insertions(+), 19 deletions(-) diff --git a/nameparser/_pipeline/_post_rules.py b/nameparser/_pipeline/_post_rules.py index aa8267fdd..53fabb2c4 100644 --- a/nameparser/_pipeline/_post_rules.py +++ b/nameparser/_pipeline/_post_rules.py @@ -689,13 +689,12 @@ def _set_apart_clauses(state: ParseState, for i in range(lo, hi): token = tokens[i] kind = set_apart_kind(token.text, lex) - if kind is None and AMBIGUOUS_ACRONYM_TAG in token.tags \ + if kind == "member" or kind is None \ + and AMBIGUOUS_ACRONYM_TAG in token.tags \ and settled_by_case(token, state.one_case): - kind = "member" - if kind is None: - break - if kind == "member": kind, member = "suffix", True + elif kind is None: + break kinds.append(kind) else: # a member is settled after the name only (rules/S13), and @@ -703,8 +702,13 @@ def _set_apart_clauses(state: ParseState, # comma part stays the nickname if member and not 0 <= last_name < lo: continue - if any(reads_as_initials(w, kind, state.one_case, lex) - for w, kind in zip(words, kinds)): + # a name word after the clause in its comma part stands + # behind the last name word, so a clause after the name skips + # the walk; a title is never initials, asked inline + if last_name >= hi and any( + kind != "title" + and reads_as_initials(w, kind, state.one_case, lex) + for w, kind in zip(words, kinds)): if ahead is None: ahead = _name_ahead(state, tokens) if ahead[hi - 1]: diff --git a/nameparser/_pipeline/_vocab.py b/nameparser/_pipeline/_vocab.py index 7741df332..2fb6e14a0 100644 --- a/nameparser/_pipeline/_vocab.py +++ b/nameparser/_pipeline/_vocab.py @@ -440,12 +440,18 @@ def set_apart_kind(word: str, lexicon: Lexicon, # necessary half of `dotted_member`, so no other word pays its # frame: `suffix_as_written` would read a dotted member as a suffix # wherever it stands, and reads no other member as one - if folded.replace(".", "") in sets.suffix_acronyms_ambiguous: + acronym = folded.replace(".", "") + if acronym in sets.suffix_acronyms_ambiguous: if title: return "title" return "member" if "." in bare and dotted_member( bare, lexicon) else None - suffix = suffix_as_written(folded, bare, lexicon) + # the sets and the joiners asked first, at C level, the necessary + # half of `suffix_as_written`, so a word it could never read (a + # nickname) pays no frame + suffix = ((acronym in sets.suffix_acronyms or folded in sets.suffix_words + or "-" in folded or "/" in folded) + and suffix_as_written(folded, bare, lexicon)) if title: return "dual" if suffix else "title" return "suffix" if suffix else None @@ -474,19 +480,28 @@ def reads_as_initials(word: str, kind: SetApartKind, word that is a title too only written with a period after each letter ('R.A.') or in capitals in a name of mixed case ('RA'), Title case reading as the title it also is and capitals in a name - written in one case saying nothing (S6). `one_case` is - ParseState.one_case. A joined credential counts where any of its - chunks would ('JP/MD', 'MD/PhD'), as its words written apart do.""" + written in one case saying nothing (S6); a title alone never. A + member of the ambiguous class counts as the credential it is + settled as. `one_case` is ParseState.one_case. A joined credential + counts where any of its chunks would ('JP/MD', 'MD/PhD'), as its + words written apart do: a chunk holds no joiner, so this is one + level deep.""" bare = word.strip(",") if kind == "suffix" and ("/" in bare or "-" in bare): for chunk in _JOINERS.split(bare): - ck = set_apart_kind(chunk, lexicon) - if ck is not None and ck != "title" and reads_as_initials( - chunk, "suffix" if ck == "member" else ck, one_case, - lexicon): + if _shaped_as_initials(chunk, set_apart_kind(chunk, lexicon), + one_case, lexicon): return True return False - return ((kind == "suffix" or kind == "dual" and ( + return _shaped_as_initials(bare, kind, one_case, lexicon) + + +def _shaped_as_initials(bare: str, + kind: SetApartKind | Literal["member"] | None, + one_case: bool | None, lexicon: Lexicon) -> bool: + """`reads_as_initials` for one word holding no joiner, its kind as + `set_apart_kind` gives it.""" + return ((kind == "suffix" or kind == "member" or kind == "dual" and ( _DOTTED_INITIALS.fullmatch(bare) is not None or one_case is False and bare.isupper())) and _INITIALS_SHAPE.fullmatch(bare) is not None diff --git a/tests/v2/test_layering.py b/tests/v2/test_layering.py index e0b1caa61..4712d3475 100644 --- a/tests/v2/test_layering.py +++ b/tests/v2/test_layering.py @@ -653,7 +653,9 @@ def test_the_check_finds_a_tool_read_through_an_alias( #: `_vocab` and both of `_post_rules`' copies of the dotted member #: test, 128 -> 123; its parity fix moving the initials test into #: `_vocab.reads_as_initials` and the dotted test into -#: `_vocab.dotted_member`, 123 -> 124); the tools' 19 are 17 +#: `_vocab.dotted_member`, 123 -> 124, and its simplify pass asking +#: two suffix sets inline ahead of `suffix_as_written`, 124 -> 126); +#: the tools' 19 are 17 #: attribute loads, every tool scanned, and config_diff.py's #: `sets =` binding and the roster `getattr` through it (its #: `changed_words`, which compares two lexicons set by set), which @@ -663,7 +665,7 @@ def test_the_check_finds_a_tool_read_through_an_alias( #: `_lexicon._fields_of` from 4i-b until 4i-c retired it. #: A reader added through `_sets` moves these, and they are re-taken #: then: they count what the check would refuse with the move undone. -_UNGUARDED_READS = {"library": 124, "tools": 19} +_UNGUARDED_READS = {"library": 126, "tools": 19} def test_the_check_refuses_every_read_4i_a_moved() -> None: From ed403c2f877911a5b37eff8b5f3734d357c0e134 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Thu, 5 Nov 2026 16:37:07 -0800 Subject: [PATCH 4/4] Leave a credential listed whole unsplit; record the remaining differences (#1088) The second-round code review found the parity fix splitting a credential the vocabulary lists whole: `John (NP-C) Smith` and `John (SHRM-CP) Smith` read the nickname, CP and C taken for initials, where master reads the suffix. `reads_as_initials` now splits only a word S3 joins; the 70 listed words holding a joiner read as master does in four positions (560 names), and two case rows pin it. The second-round design-docs review: rules/S1 gains an Accepted bullet for the two classes where a joined word reads otherwise than its words written apart (a member S13 does not settle, a title-too word at either title site) and the listed-whole qualifier with an `NP-C` boundary; decisions/S1 corrects the period-final move (a nickname no release read), the `(MD/PhD, MBA)` comparison (master reported it, 2.8.2 did not) and writes the grid's exact templates. The snapshot gains the five new examples; no existing entry moves. Co-Authored-By: Claude Opus 5.5 --- .../S1-bracketed-credential-is-not-a-nickname.md | 6 +++--- .../S1-bracketed-credential-is-not-a-nickname.md | 10 +++++++++- nameparser/_pipeline/_vocab.py | 10 +++++++--- tests/v2/cases.py | 14 ++++++++++++++ tests/v2/rules_budget.json | 2 +- tests/v2/test_layering.py | 6 ++++-- tools/differential/corpus_rules.jsonl | 5 +++++ tools/differential/snapshot.jsonl | 5 +++++ 8 files changed, 48 insertions(+), 10 deletions(-) diff --git a/docs/design/decisions/S1-bracketed-credential-is-not-a-nickname.md b/docs/design/decisions/S1-bracketed-credential-is-not-a-nickname.md index a90eb7386..99b22410d 100644 --- a/docs/design/decisions/S1-bracketed-credential-is-not-a-nickname.md +++ b/docs/design/decisions/S1-bracketed-credential-is-not-a-nickname.md @@ -20,6 +20,6 @@ Elsewhere: an ambiguous member alone in the brackets staying a nickname, reporte ACCEPTED COSTS (rules/S1's Accepted bullets): a clause between name words reads otherwise than the same words bare (`W. Russell Jr. Withers` keeps middle 'Russell Jr.'); a short title word a writer's data uses as a tag reads as the title (FEC's state codes, `CASHION, JOHN(ST)` and six more entries holding `(ST)`, two holding `(ME)`, by the same recipe); a title-only clause behind a credential keeps its pair's reading even where it holds a title of the name and loses the period escape it took (`John Smith, MD (Dr.)` → nickname 'Dr.'; `Louise Dye, PhD (Professor)` stays the nickname it was); a set-apart clause in a name of no word of its own leaves the name empty (`(MPH)` → suffix 'MPH', where the escape had made it given 'MPH', H4); and `Col. (Ret.) Smith` reads suffix 'Ret.' where 1.4.0 through 2.8.0 read title 'Col. Ret.' (tests/test_suffixes.py's two "known limitation" rows, `Lon (Jr.) Williams` now suffix 'Jr.', are re-pinned). MEASURED 2026-11-04 with `snapshot.py --local --base ` (head package digest 9b70692c0bafee65): ctgov-officials 26 entries move, fec-individuals 16, psc-individuals 10, orcid-researchers 79; govuk-people, ukec-individuals, cz-candidates and dof-signatures none. By class: a bracketed title read as the title where it read as the nickname (`Dorothee Auer (Principal Investigator)`, `Prof (Dr) Ashish Gadekar` → title 'Prof Dr', `Mary (Rev Mrs) Fosu`; FEC's state codes `(ST)` and `(ME)`, the accepted cost); a credential run read as the suffix where it read as the nickname (`Brian Chirombo (MBChB, MPH)`, `F.L.J. Visseren, Professor (MD PhD)`); a bracketed generation or credential between name words ending at its brackets (`Mr Hugh (Snr) Boyle` → suffix 'Snr' where middle, `Abebe Alemu (PhD) Balcha` → family 'Balcha', `Dr (Phd) Alan Green` → given 'Alan'); credential initials inside a name moving from the middle name or the suffix run to the nickname (`FOX, (JP)JIMMY`); two ORCID nickname clauses gaining the member report above; and FEC's `TOBIN, MARY (V)` losing the suffix-or-name report #877 gives a bare letter ending the given part after a comma, which master gave it because its extract stage drops the brackets from content that is suffix-shaped, reading it bare: the clause is set apart in post_rules, after assign's walk reports, and a letter its writer bracketed is set apart from the name rather than a middle initial in doubt, so rules/C9 names it among the slots that settle the letter (the first measurement, against c6207549 before #877 merged, counted FEC 16). CLDR agreement: one Czech rendering moves, `Machová, Alexandra Zuzana (paní, Ph.D.)`, its given names agreeing now and its title gaining 'paní', which CLDR leaves out. - 2026-11-05 (Derek), #1088 — A JOINED CREDENTIAL READS AS ITS WORDS WRITTEN APART. `John Smith (MD, PhD)` read suffix 'MD, PhD' under #792, but `John Smith (MD/PhD)` and `John Smith (MD-PhD)` read as nicknames, though S3 reads the bare `John Smith MD/PhD` as the suffix: the set-apart word asked its own copy of the unambiguous suffix sets, which S3's joiner arm is not in. #1084's simplify pass proposed asking `suffix_as_written`, the bare word's question, and refused it as a behavior change, since that predicate also reads a dotted member of the ambiguous class (`M.A.`) as a suffix, which S13 settles only after the name. DECIDED (Derek), as proposed: the word asks the bare word's question, a member of the ambiguous class asked first and left to S13, whose position test is unchanged. The member's dotted test, which post_rules had copied twice, is one predicate now (`_vocab.dotted_member`), asked by the settling and by the report alike, so the report on a member left the nickname leaves off a comma the clause keeps, as the settling did (rules/N1): `Jean (M.A,) Smith` reports suffix-or-nickname, as `Jean (M.A) Smith` does. The interior-period half of S3 (`(Msc.Ed)`, a different predicate whose any-chunk test is looser than the joiner's every-chunk) was left out, for an issue drafted the same day. - THE INITIALS EXCEPT, decided after review: the first build set a joined credential apart before a name word too, where its words written apart stay the nickname under S1's initials Except, which asked the whole word, and a joined word never has the shape of initials (`John (MD/PhD) Smith` and `Justin (JP/MD) Stearns` → suffix, against `John (MD, PhD) Smith` and `Justin (JP, MD) Stearns` → nickname; the code, comment and design-docs reviews each found it, the last by the two-spelling invariant over 3,360 pairs). OPTIONS PUT: (A) the Except asks each chunk, so the two spellings read alike; (B) keep the divergence as an accepted cost. DECIDED (Derek): A. A joined credential counts as initials where any of its chunks would, as its words written apart do (`JP/MD`, `MD/PhD`, `M.D./Ph.D.`), and one whose chunks are generation words does not (`John (Jr/Sr) Smith` → suffix, as `(Jr, Sr)` reads). tests/v2/pipeline/test_post_rules.py holds the two spellings alike over a grid of plain credential pairs in eleven shapes under the three orders; with the chunk loop off, 540 of its 2,310 rows read apart. Where the spelling written apart reads by another rule the joined word does not meet, the two still differ, each by that rule: a word that is a title too reads as the title alone in a name with no word of its own (`(MD, MS)` → title 'MS', `(MD/MS)` → suffix 'MD/MS'), and a clause ending in a member written with its periods takes the period escape written apart (`John (MD, M.A.) Smith` → suffix 'MD, M.A. Smith', `John (MD/M.A.) Smith` → nickname). - WHAT IT MOVES, and accepted: a joined credential ending in a period is set apart before the period escape is asked, which had read it bare, so `Smith (M.D./Ph.D.), John` → suffix as 2.8 read it through the escape, `Jack (M.D./Ph.D.)` → family 'Jack', as `Jack (PhD)` reads, where 2.8 read given 'Jack' and reported given-or-family, and `John (M.D./Ph.D.) Smith`, which the initials Except declines, keeps its pair's reading rather than the escape (nickname, where 1.4.0 through 2.8 read middle 'M.D./Ph.D.'). A clause the joined word completes is set apart whole, so a member beside it that its capitals settle no longer reports (`John Smith (MD/PhD, MBA)` → suffix 'MD/PhD, MBA', where 2.8 read the nickname and reported suffix-or-nickname). Not moved: a joined word with a chunk that is no credential or is a bare member (`MD/Bob`, `MA/PhD`), a hyphenated name (`Smith-Jones`, `Ae-Ra`), single-letter numerals (`I-V`) and a dangling joiner (`MD/`), each as S3 already refuses it bare. - MEASURED 2026-11-05 against master 4109b026 checked out in a detached worktree: the committed snapshot moves none of its 6,945 entries (6,736 distinct names), and none of the 181 distinct names holding a pair of `DEFAULT_NICKNAME_DELIMITERS` has a word joined by a slash or a hyphen inside one, so the zero says the snapshot could not see the shape (recipe: for each name and each pair, the text between an opener and the next closer after it, matched against `\S[-/]\S`); the rule's nine new examples join the snapshot through corpus_rules.jsonl in the same change, recorded as new entries. A grid of 30 clause words (thirteen that move: the joined credentials `MD/PhD`, `MD-PhD`, `Md-PhD`, `md/phd`, `MD/PhD/MPH`, `M.A./PhD`, `Jr/Sr`, `MD/PhD,`, `M.D./Ph.D.`, and the clauses `MD/PhD, MBA` and `PhD, MD/MPH` holding one, and the kept-comma members `M.A,` and `M.A.,`; seventeen that do not: S3's refusals bare, a bare member as a chunk (`PhD/MBA`, `MA/PhD`, `JD/MBA`), a chunk not written with its periods (`M.A/PhD`), an unlisted chunk (`RN-BC`, `CPA/PFS`, `XY/ZW`), a name or a title as a chunk (`MD/Bob`, `Bob/MD`, `Dr/PhD`), a dangling joiner (`MD/`, `/PhD`), single-letter numerals (`I-V`) and a hyphenated name (`Smith-Jones`, `Ae-Ra`), and the members without a comma, `M.A` and `MA`) in 14 shapes (after the name, inside it, before it, alone, after and before a family comma, before and after a suffix comma, a name in capitals, a lone name word, an empty head before a comma, before a suffix comma with nothing ahead, in quotes, and after a two-word Korean name), each under GIVEN_FIRST, FAMILY_FIRST and FAMILY_FIRST_GIVEN_LAST, 1,260 names: 399 move, 133 under each order, every move a joined credential read as the suffix or the member report (447, 149 under each order, before the initials Except asked each chunk). The comparator is the base tree's own readings of the same grid, parsed with each tree's `nameparser.__file__` asserted. In the local corpora (`snapshot.py --local --base `, on the first build): ctgov-officials 3 entries move, `Franck Pages, Professor (MD-PHD)` and `Franck Pagès, Professor (MD-PHD)` and `Philippe PC Courtet, Professor(MD-PhD)` → suffix where nickname; fec-individuals 1, `FRANTZ (CAPT USN-RET), JOSEPH` → title 'CAPT', suffix 'USN-RET' where nickname 'CAPT USN-RET'; psc-individuals and orcid-researchers none. The initials Except only returns a joined word before a name word to the nickname master read, and none of the four stands there; each was parsed again on the final tree and reads as listed. + THE INITIALS EXCEPT, decided after review: the first build set a joined credential apart before a name word too, where its words written apart stay the nickname under S1's initials Except, which asked the whole word, and a joined word never has the shape of initials (`John (MD/PhD) Smith` and `Justin (JP/MD) Stearns` → suffix, against `John (MD, PhD) Smith` and `Justin (JP, MD) Stearns` → nickname; the code, comment and design-docs reviews each found it, the last by the two-spelling invariant over 3,360 pairs). OPTIONS PUT: (A) the Except asks each chunk, so the two spellings read alike; (B) keep the divergence as an accepted cost. DECIDED (Derek): A. A joined credential counts as initials where any of its chunks would, as its words written apart do (`JP/MD`, `MD/PhD`, `M.D./Ph.D.`), and one whose chunks are generation words does not (`John (Jr/Sr) Smith` → suffix, as `(Jr, Sr)` reads). A credential the vocabulary lists whole is one word, not two S3 joins, and is never split (`John (NP-C) Smith` and `John (SHRM-CP) Smith` → suffix, as master reads them): the second-round code review found the first parity fix splitting it, `CP` and `C` reading as initials, and a sweep of the 70 listed words holding a joiner (two spellings each, in four positions, 560 names) reads every one as master does. tests/v2/pipeline/test_post_rules.py holds the two spellings alike over a grid of plain credential pairs in eleven shapes under the three orders; with the chunk loop off, 540 of its 2,310 rows read apart. Where the spelling written apart reads by another rule the joined word does not meet, the two still differ, each by that rule, and rules/S1 records the first two as Accepted: a word that is a title too reads as the title in front of the name's first word or in a name with no word of its own (`(PhD, Sr) John Smith` → title 'Sr', `(PhD/Sr) John Smith` → suffix 'PhD/Sr'; `(MD, MS)` → title 'MS', `(MD/MS)` → suffix 'MD/MS'); a member of the ambiguous class written as its own word is settled by S13's position test, which a joined word S3 reads whole does not take (`(M.A., PhD)` and `Smith (M.A., PhD), John` → nickname, reported, `(M.A./PhD)` and `Smith (M.A./PhD), John` → suffix, where master and 2.8.2 read both spellings as the nickname; the second-round design-docs review found it over 40 such pairs); and a clause ending in a member written with its periods takes the period escape written apart (`John (MD, M.A.) Smith` → suffix 'MD, M.A. Smith', `John (MD/M.A.) Smith` → nickname). + WHAT IT MOVES, and accepted: a joined credential ending in a period is set apart before the period escape is asked, which had read it bare, so `Smith (M.D./Ph.D.), John` → suffix as 2.8 read it through the escape, `Jack (M.D./Ph.D.)` → family 'Jack', as `Jack (PhD)` reads, where 2.8 read given 'Jack' and reported given-or-family, and a period-final joined word before a name word, which the initials Except declines, keeps its pair's reading rather than the escape, a nickname no release read (`John (M.D./Ph.D.) Smith`, where 1.4.0 through 2.8 read middle 'M.D./Ph.D.', and `(M.D./Ph.D.) John Smith`, where they read given 'M.D./Ph.D.'). A clause the joined word completes is set apart whole, so a member beside it that its capitals settle no longer reports (`John Smith (MD/PhD, MBA)` → suffix 'MD/PhD, MBA', where master read the nickname and reported suffix-or-nickname, `mba` having been marked ambiguous on 2026-11-03; 1.4.0 and 2.8.2 read the nickname with no report). Not moved: a joined word with a chunk that is no credential or is a bare member (`MD/Bob`, `MA/PhD`), a hyphenated name (`Smith-Jones`, `Ae-Ra`), single-letter numerals (`I-V`) and a dangling joiner (`MD/`), each as S3 already refuses it bare. + MEASURED 2026-11-05 against master 4109b026 checked out in a detached worktree: the committed snapshot moves none of its 6,945 entries (6,736 distinct names), and none of the 181 distinct names holding a pair of `DEFAULT_NICKNAME_DELIMITERS` has a word joined by a slash or a hyphen inside one, so the zero says the snapshot could not see the shape (recipe: for each name and each pair, the text between an opener and the next closer after it, matched against `\S[-/]\S`); the rule's fourteen new examples join the snapshot through corpus_rules.jsonl in the same change, recorded as new entries. A grid of 30 clause words (thirteen that move: the joined credentials `MD/PhD`, `MD-PhD`, `Md-PhD`, `md/phd`, `MD/PhD/MPH`, `M.A./PhD`, `Jr/Sr`, `MD/PhD,`, `M.D./Ph.D.`, and the clauses `MD/PhD, MBA` and `PhD, MD/MPH` holding one, and the kept-comma members `M.A,` and `M.A.,`; seventeen that do not: S3's refusals bare, a bare member as a chunk (`PhD/MBA`, `MA/PhD`, `JD/MBA`), a chunk not written with its periods (`M.A/PhD`), an unlisted chunk (`RN-BC`, `CPA/PFS`, `XY/ZW`), a name or a title as a chunk (`MD/Bob`, `Bob/MD`, `Dr/PhD`), a dangling joiner (`MD/`, `/PhD`), single-letter numerals (`I-V`) and a hyphenated name (`Smith-Jones`, `Ae-Ra`), and the members without a comma, `M.A` and `MA`) in 14 shapes, the templates exactly `John Smith (X)`, `John (X) Smith`, `(X) John Smith`, `(X)`, `Smith, John (X)`, `Smith (X), John`, `John Smith (X), Jr.`, `John Smith, Jr. (X)`, `JOHN SMITH (X)`, `Jack (X)`, `Smith, (X)`, `(X), PhD`, `John Smith "X"` and `Kim Min (X)`, each parsed by `Parser()` and by `Parser(policy=Policy(name_order=...))` under FAMILY_FIRST and FAMILY_FIRST_GIVEN_LAST, a name moving where `as_dict()` or its ambiguity kinds differ, 1,260 names: 399 move, 133 under each order, every move a joined credential read as the suffix, a period-final joined word before a name word read as the nickname (two under each order), or the member report (447, 149 under each order, before the initials Except asked each chunk). The comparator is the base tree's own readings of the same grid, parsed with each tree's `nameparser.__file__` asserted. In the local corpora (`snapshot.py --local --base `, on the first build): ctgov-officials 3 entries move, `Franck Pages, Professor (MD-PHD)` and `Franck Pagès, Professor (MD-PHD)` and `Philippe PC Courtet, Professor(MD-PhD)` → suffix where nickname; fec-individuals 1, `FRANTZ (CAPT USN-RET), JOSEPH` → title 'CAPT', suffix 'USN-RET' where nickname 'CAPT USN-RET'; psc-individuals and orcid-researchers none. The initials Except reaches only a joined word before a name word of its comma part, and none of the four stands there; each was parsed again on the final tree and reads as listed. diff --git a/docs/design/rules/S1-bracketed-credential-is-not-a-nickname.md b/docs/design/rules/S1-bracketed-credential-is-not-a-nickname.md index 7cb912fce..2fa01274e 100644 --- a/docs/design/rules/S1-bracketed-credential-is-not-a-nickname.md +++ b/docs/design/rules/S1-bracketed-credential-is-not-a-nickname.md @@ -69,7 +69,7 @@ "Alex Green, FRCS (SN)" → nickname="SN" "John Smith, MD (Sr)" → suffix="MD Sr" · boundary ``` -- **A credential written as initials before a name word of its comma part** keeps its pair's reading: one or two letters with or without a period, or single letters each with its period, there are more likely the initials the person goes by, S13's reason for a member inside the name. A credential written as initials before a name word stays the nickname, and so does any word of its clause; a word that is a title too counts only written with a period after each letter, or in capitals in a name of mixed case, capitals in a name written in one case saying nothing (S6), and a generation word not at all; a credential joined by a slash or a hyphen counts where any of its chunks would, as its words written apart do. +- **A credential written as initials before a name word of its comma part** keeps its pair's reading: one or two letters with or without a period, or single letters each with its period, there are more likely the initials the person goes by, S13's reason for a member inside the name. A credential written as initials before a name word stays the nickname, and so does any word of its clause; a word that is a title too counts only written with a period after each letter, or in capitals in a name of mixed case, capitals in a name written in one case saying nothing (S6), and a generation word not at all; a credential S3 reads through chunks joined by a slash or a hyphen counts where any of its chunks would, as its words written apart do, and one the vocabulary lists whole is one word. ``` "Justin (JP) Stearns" → nickname="JP" "Justin (R.N.) Stearns" → nickname="R.N." @@ -82,6 +82,7 @@ "ROBERT (RA) ALLEN" → suffix="RA" · boundary "John (Sr) Smith" → suffix="Sr" · boundary "John (Jr/Sr) Smith" → suffix="Jr/Sr" · boundary + "John (NP-C) Smith" → suffix="NP-C" · boundary "Smith (MD), John" → suffix="MD" · boundary ``` - **A clause holding a word the vocabulary reads as a name, or does not know**, is no such clause, and reads as its pair reads it, a nickname in a nickname pair. A member of the ambiguous class alone is S13's. @@ -125,6 +126,13 @@ "MPH" → given="MPH" "(Dr), John" → family="John" ``` +- A credential joined by a slash or a hyphen is one word, so it reads otherwise than its words written apart where one of them reads by a rule of its own: a member of the ambiguous class whose position S13 does not settle, and a word that is a title too where it would be the title. + ``` + "(M.A./PhD)" → suffix="M.A./PhD" + "(M.A., PhD)" → nickname="M.A., PhD" + "(PhD/Sr) John Smith" → suffix="PhD/Sr" + "(PhD, Sr) John Smith" → title="Sr" + ``` - A qualifier written after a title in front of the name is the suffix it is in the vocabulary. ``` "Col. (Ret.) Smith" → suffix="Ret." diff --git a/nameparser/_pipeline/_vocab.py b/nameparser/_pipeline/_vocab.py index 2fb6e14a0..fce84b18c 100644 --- a/nameparser/_pipeline/_vocab.py +++ b/nameparser/_pipeline/_vocab.py @@ -484,10 +484,14 @@ def reads_as_initials(word: str, kind: SetApartKind, member of the ambiguous class counts as the credential it is settled as. `one_case` is ParseState.one_case. A joined credential counts where any of its chunks would ('JP/MD', 'MD/PhD'), as its - words written apart do: a chunk holds no joiner, so this is one - level deep.""" + words written apart do, and one the vocabulary lists whole never + ('NP-C'): a chunk holds no joiner, so this is one level deep.""" bare = word.strip(",") - if kind == "suffix" and ("/" in bare or "-" in bare): + # a credential the vocabulary lists whole ('NP-C', 'SHRM-CP') is one + # word, not two S3 joins, and has no other spelling written apart + if kind == "suffix" and ("/" in bare or "-" in bare) and not ( + (folded := _normalize(bare)) in lexicon._sets.suffix_words + or folded.replace(".", "") in lexicon._sets.suffix_acronyms): for chunk in _JOINERS.split(bare): if _shaped_as_initials(chunk, set_apart_kind(chunk, lexicon), one_case, lexicon): diff --git a/tests/v2/cases.py b/tests/v2/cases.py index dcf0b4295..ca264208f 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -9921,6 +9921,20 @@ def _check_cjk_shape_purity(self) -> None: 'Smith (M.A.), John', {'given': 'John', 'family': 'Smith', 'suffix': 'M.A.'}, notes="as the row above: the period escape, not S1's set-apart reading, which would make it a reported nickname before the comma"), + Case("paren_listed_hyphenated_credential_inside_the_name_set_apart", + "John (NP-C) Smith", + {"given": "John", "family": "Smith", "suffix": "NP-C"}, + classification="fix(#792)", + notes="a credential listed whole is one word, not two joined: " + "the initials Except asks its chunks only of a word S3 " + "joins, and 'NP, C' is no spelling of it (#1088's second " + "review, whose first fix split it, reading the nickname)"), + Case("paren_listed_hyphenated_credential_before_a_name_set_apart", + "John (SHRM-CP) Smith", + {"given": "John", "family": "Smith", "suffix": "SHRM-CP"}, + classification="fix(#792)", + notes="as the row above; CP is a listed two-letter acronym, so a " + "chunk test reads it as initials"), Case("paren_joined_credential_and_name_stays_the_nickname", "John Smith (MD/Bob)", {"given": "John", "family": "Smith", "nickname": "MD/Bob"}, diff --git a/tests/v2/rules_budget.json b/tests/v2/rules_budget.json index 8cc33fc9e..6d3c7c988 100644 --- a/tests/v2/rules_budget.json +++ b/tests/v2/rules_budget.json @@ -64,7 +64,7 @@ "R9": {"words": 397, "then_words": 122, "because_sentences": 3, "except": 5, "precedence": 0, "accepted": 13, "accepted_words": 477}, "R10": {"words": 168, "then_words": 26, "because_sentences": 2, "except": 2, "precedence": 0, "accepted": 6, "accepted_words": 160}, "R11": {"words": 226, "then_words": 42, "because_sentences": 3, "except": 2, "precedence": 0, "accepted": 5, "accepted_words": 195}, - "S1": {"words": 736, "then_words": 248, "because_sentences": 3, "except": 6, "precedence": 3, "accepted": 5, "accepted_words": 173}, + "S1": {"words": 749, "then_words": 248, "because_sentences": 3, "except": 6, "precedence": 3, "accepted": 6, "accepted_words": 231}, "S2": {"words": 658, "then_words": 120, "because_sentences": 1, "except": 0, "precedence": 7, "accepted": 3, "accepted_words": 222}, "S3": {"words": 960, "then_words": 398, "because_sentences": 3, "except": 4, "precedence": 5, "accepted": 6, "accepted_words": 357}, "S4": {"words": 306, "then_words": 164, "because_sentences": 2, "except": 0, "precedence": 3, "accepted": 0, "accepted_words": 0}, diff --git a/tests/v2/test_layering.py b/tests/v2/test_layering.py index 4712d3475..14f9953c7 100644 --- a/tests/v2/test_layering.py +++ b/tests/v2/test_layering.py @@ -654,7 +654,9 @@ def test_the_check_finds_a_tool_read_through_an_alias( #: test, 128 -> 123; its parity fix moving the initials test into #: `_vocab.reads_as_initials` and the dotted test into #: `_vocab.dotted_member`, 123 -> 124, and its simplify pass asking -#: two suffix sets inline ahead of `suffix_as_written`, 124 -> 126); +#: two suffix sets inline ahead of `suffix_as_written`, 124 -> 126, and +#: `reads_as_initials` leaving a credential listed whole unsplit, two +#: more, 126 -> 128); #: the tools' 19 are 17 #: attribute loads, every tool scanned, and config_diff.py's #: `sets =` binding and the roster `getattr` through it (its @@ -665,7 +667,7 @@ def test_the_check_finds_a_tool_read_through_an_alias( #: `_lexicon._fields_of` from 4i-b until 4i-c retired it. #: A reader added through `_sets` moves these, and they are re-taken #: then: they count what the check would refuse with the move undone. -_UNGUARDED_READS = {"library": 126, "tools": 19} +_UNGUARDED_READS = {"library": 128, "tools": 19} def test_the_check_refuses_every_read_4i_a_moved() -> None: diff --git a/tools/differential/corpus_rules.jsonl b/tools/differential/corpus_rules.jsonl index 970f1091b..a054d57d5 100644 --- a/tools/differential/corpus_rules.jsonl +++ b/tools/differential/corpus_rules.jsonl @@ -8,8 +8,12 @@ "(Lt)" "(Lt) John Smith" "(M.A)" +"(M.A., PhD)" +"(M.A./PhD)" "(MA) John Smith" "(MPH)" +"(PhD, Sr) John Smith" +"(PhD/Sr) John Smith" ", Ana García López" ", Dean" ", Dr. John" @@ -483,6 +487,7 @@ "John (Jack) Smith" "John (Jr/Sr) Smith" "John (MD-PhD) Smith" +"John (NP-C) Smith" "John (Sr) Smith" "John (Sr.) Smith" "John . Smith" diff --git a/tools/differential/snapshot.jsonl b/tools/differential/snapshot.jsonl index 7facf832d..3aa57cd8f 100644 --- a/tools/differential/snapshot.jsonl +++ b/tools/differential/snapshot.jsonl @@ -15,8 +15,12 @@ {"name": "(Lt)", "title": "Lt"} {"name": "(Lt) John Smith", "title": "Lt", "given": "John", "family": "Smith", "initials": "J. S."} {"name": "(M.A)", "nickname": "M.A", "ambiguities": ["SUFFIX_OR_NICKNAME"], "alternatives": ["SUFFIX_OR_NICKNAME: credential as suffix (vocabulary)"]} +{"name": "(M.A., PhD)", "nickname": "M.A., PhD", "ambiguities": ["SUFFIX_OR_NICKNAME"], "alternatives": ["SUFFIX_OR_NICKNAME: credential as suffix (vocabulary)"]} +{"name": "(M.A./PhD)", "suffix": "M.A./PhD"} {"name": "(MA) John Smith", "given": "John", "family": "Smith", "nickname": "MA", "ambiguities": ["SUFFIX_OR_NICKNAME"], "alternatives": ["SUFFIX_OR_NICKNAME: credential as suffix (vocabulary)"], "initials": "J. S.", "capitalized_force": "John \"Ma\" Smith"} {"name": "(MPH)", "suffix": "MPH"} +{"name": "(PhD, Sr) John Smith", "title": "Sr", "given": "John", "family": "Smith", "suffix": "PhD", "initials": "J. S."} +{"name": "(PhD/Sr) John Smith", "given": "John", "family": "Smith", "suffix": "PhD/Sr", "initials": "J. S."} {"name": "(김민준.) John Smith", "given": "김", "middle": "민준. John", "family": "Smith", "initials": "J. S."} {"name": ","} {"name": ", Ana García López", "given": "Ana", "middle": "García López", "initials": "A. G. L."} @@ -1754,6 +1758,7 @@ {"name": "John (Jack) Smith", "given": "John", "family": "Smith", "nickname": "Jack", "initials": "J. S."} {"name": "John (Jr/Sr) Smith", "given": "John", "family": "Smith", "suffix": "Jr/Sr", "initials": "J. S."} {"name": "John (MD-PhD) Smith", "given": "John", "family": "Smith", "nickname": "MD-PhD", "initials": "J. S.", "capitalized_force": "John \"Md-PhD\" Smith"} +{"name": "John (NP-C) Smith", "given": "John", "family": "Smith", "suffix": "NP-C", "initials": "J. S."} {"name": "John (Sr) Smith", "given": "John", "family": "Smith", "suffix": "Sr", "initials": "J. S.", "generation": "Sr"} {"name": "John (Sr.) Smith", "given": "John", "family": "Smith", "suffix": "Sr.", "initials": "J. S.", "generation": "Sr."} {"name": "John . Smith", "given": "John", "middle": ".", "family": "Smith", "initials": "J. .. S."}