diff --git a/AGENTS.md b/AGENTS.md index 789c011f8..f0eb0891d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -30,7 +30,7 @@ Three committed contributor docs carry the parser's normative rules and their re **Counting claims.** A bare count in prose is either an assertion or a liability, keyed by who observes its staleness: asserted counts (a test holds the number) fail CI at change time — the useful kind; dated snapshots ("51 sites at spec time") cannot go stale; standing present-tense prose counts are the forbidden class — promote to an assertion, add a date, or state the invariant and let a test count. After changing how many times something runs, sweep for counts, not for the thing's name. -**Release-log claims.** Quantified or universal behavior claims in release bullets must come from the snapshot's diff since the last release, be verified against rules examples, or -- for a view the snapshot cannot see -- carry a recompute recipe stored with the design entry the bullet cites; never write one from memory. `uv run python tools/differential/snapshot.py --since v` prints every corpus name the release moved, grouped by changed field; v2.4.1 is the first tag carrying a snapshot, and for an older release `compare.py --baseline ` prints the same report against the PyPI wheel, closing with the entries compared and how many moved names are Latin-only. Both say how many moved names only `corpus_cldr.jsonl` holds. The snapshot records the seven role fields, ambiguity kinds, `initials()`, the two case-repair views (`capitalized()` and `capitalized(force=True)`, each only where it changes the text, #826) and, since 3.0, the suffix's `generation` and `credentials` views (where the suffix holds a generation, rules/R8) and the `patronymic` and `surname2` views (each where it is not empty, rules/R9 and R10; the snapshot parses the default policy, so no name fills `surname2`, which only `two_family_names` reads) for every name, so an initials, case, generation or patronymic count is read off it directly -- with two limits. A case view holds the whole repaired name, so in a name repair changes, a role move that changes the rendering moves the view too, with or without a word recased: read a view move that comes with a role move by its words' case, not by whether it moved (#808's `John Smith, otr/l` moved both, `'John Smith OTR/L'` -> `'Otr/L John Smith'`). And `--since` a tag whose snapshot predates #826 (v2.5.0 and earlier) does not compare the views at all, so 2.6.0's case claims come from `compare.py --baseline 2.5.0`. `corpus_cldr.jsonl`'s fictional renderings are in it: count them out of a release figure (both tools print how many moved names only that file holds), or say the figure includes them. A render view the snapshot does not record -- `str()` under a custom format, `initials()` under a custom spec -- stays invisible to it, and for those the first two sources cannot reach a claim: the snapshot is byte-identical across the change, and an example line witnesses an output without counting anything. The DEFAULT `str()` is mostly invisible too, the snapshot seeing it only through a case view where repair changes the text: #754's render defaults moved `str()` for 1,034 corpus entries and the case views for 159, so a change to the default rendering is measured by rendering every entry in both trees (decisions/R6's 2026-10-29 entry holds the recipe). A recipe names the corpus files, the policy sweep, and -- the part that is easy to omit and fatal -- THE COMPARATOR, which must be something the shipped tree is not: #408's first recipe said to compare `initials()` against a folded-first partition, which is what `initials()` now IS, so it reproduced 0 where the bullet claimed 660 and was the only stated provenance for the number. Run the recipe as written before shipping the bullet. Cross-version numbers (a released wheel, the pre-change tree) are dated snapshots under Counting claims, since nothing in the repository re-runs them. The same standard binds decisions entries and PR bodies: a claim that a change restores the released reading is measured over the whole moved population against the wheel, never spot-checked (#625 claimed its 774 moves landed on the 1.4.0 and 2.3.0 readings from three names; measured, the tree matched 2.3.0 on 0 of 258 moved texts, master on 152), and a population size is re-counted from its recipe, never copied from another entry (the same entry carried #624's 137,810 for a grid of 395,148). Code comments citing rule IDs follow the excerpt discipline, checked by tests/v2/test_doc_citations.py: a rule's quotable text is its fields' prose (When, Tolerated, Then and its bullets, Because, and the Except, Precedence and Accepted bullets), and its fences and the footer below `---`, where the `Signal:` line stands, are not; `rules/H` names the H section, whose Background is quotable; a decisions citation may paraphrase, but what it quotes must be verbatim; and a reference with no quote must still name something that exists. The check's reach is a quote within four words of the ID or chained to one that is: a quote further off reads exactly like a name written in double quotes, so it is NOT checked, and the second review of #632 found 14 such quotes stale by hand. Keep a quote beside its ID. Quotes of AGENTS.md itself are not checked at all. **A citation comment holds three things: the quote, a line saying which clause this code implements, and `decisions/X (#N)` pointers to the entries that hold the history.** A code-local reason a reader of that line needs may stay beside them (why an index is safe to keep, why a test runs last), but issue history, measurements and restated examples belong in decisions/, where they are dated and kept current; in a comment they are a second copy that goes stale where nobody looks. #760 trimmed five blocks of 59 to 100 lines that had grown that way (`_post_rules._mark_suffix_entries`, group's P3 both-sides test, its S2/P2/P4 chain head and S4 chain emitter, and the honorific peel's scope in `_script_segment`), moving into dated decisions bullets what those entries lacked. Two of the moved claims no longer held: a policy example that had stopped reaching its case, and a credential said to land as a title that lands as a suffix. +**Release-log claims.** Quantified or universal behavior claims in release bullets must come from the snapshot's diff since the last release, be verified against rules examples, or -- for a view the snapshot cannot see -- carry a recompute recipe stored with the design entry the bullet cites; never write one from memory. `uv run python tools/differential/snapshot.py --since v` prints every corpus name the release moved, grouped by changed field; v2.4.1 is the first tag carrying a snapshot, and for an older release `compare.py --baseline ` prints the same report against the PyPI wheel, closing with the entries compared and how many moved names are Latin-only. Both say how many moved names only `corpus_cldr.jsonl` holds. The snapshot records the seven role fields, ambiguity kinds, `initials()`, the two case-repair views (`capitalized()` and `capitalized(force=True)`, each only where it changes the text, #826) and, since 3.0, the suffix's `generation` and `credentials` views (where the suffix holds a generation, rules/R8), its `honorifics` view (where it is not empty, rules/R8, #326) and the `patronymic` and `surname2` views (each where it is not empty, rules/R9 and R10; the snapshot parses the default policy, so no name fills `surname2`, which only `two_family_names` reads) for every name, so an initials, case, generation or patronymic count is read off it directly -- with two limits. A case view holds the whole repaired name, so in a name repair changes, a role move that changes the rendering moves the view too, with or without a word recased: read a view move that comes with a role move by its words' case, not by whether it moved (#808's `John Smith, otr/l` moved both, `'John Smith OTR/L'` -> `'Otr/L John Smith'`). And `--since` a tag whose snapshot predates #826 (v2.5.0 and earlier) does not compare the views at all, so 2.6.0's case claims come from `compare.py --baseline 2.5.0`. `corpus_cldr.jsonl`'s fictional renderings are in it: count them out of a release figure (both tools print how many moved names only that file holds), or say the figure includes them. A render view the snapshot does not record -- `str()` under a custom format, `initials()` under a custom spec -- stays invisible to it, and for those the first two sources cannot reach a claim: the snapshot is byte-identical across the change, and an example line witnesses an output without counting anything. The DEFAULT `str()` is mostly invisible too, the snapshot seeing it only through a case view where repair changes the text: #754's render defaults moved `str()` for 1,034 corpus entries and the case views for 159, so a change to the default rendering is measured by rendering every entry in both trees (decisions/R6's 2026-10-29 entry holds the recipe). A recipe names the corpus files, the policy sweep, and -- the part that is easy to omit and fatal -- THE COMPARATOR, which must be something the shipped tree is not: #408's first recipe said to compare `initials()` against a folded-first partition, which is what `initials()` now IS, so it reproduced 0 where the bullet claimed 660 and was the only stated provenance for the number. Run the recipe as written before shipping the bullet. Cross-version numbers (a released wheel, the pre-change tree) are dated snapshots under Counting claims, since nothing in the repository re-runs them. The same standard binds decisions entries and PR bodies: a claim that a change restores the released reading is measured over the whole moved population against the wheel, never spot-checked (#625 claimed its 774 moves landed on the 1.4.0 and 2.3.0 readings from three names; measured, the tree matched 2.3.0 on 0 of 258 moved texts, master on 152), and a population size is re-counted from its recipe, never copied from another entry (the same entry carried #624's 137,810 for a grid of 395,148). Code comments citing rule IDs follow the excerpt discipline, checked by tests/v2/test_doc_citations.py: a rule's quotable text is its fields' prose (When, Tolerated, Then and its bullets, Because, and the Except, Precedence and Accepted bullets), and its fences and the footer below `---`, where the `Signal:` line stands, are not; `rules/H` names the H section, whose Background is quotable; a decisions citation may paraphrase, but what it quotes must be verbatim; and a reference with no quote must still name something that exists. The check's reach is a quote within four words of the ID or chained to one that is: a quote further off reads exactly like a name written in double quotes, so it is NOT checked, and the second review of #632 found 14 such quotes stale by hand. Keep a quote beside its ID. Quotes of AGENTS.md itself are not checked at all. **A citation comment holds three things: the quote, a line saying which clause this code implements, and `decisions/X (#N)` pointers to the entries that hold the history.** A code-local reason a reader of that line needs may stay beside them (why an index is safe to keep, why a test runs last), but issue history, measurements and restated examples belong in decisions/, where they are dated and kept current; in a comment they are a second copy that goes stale where nobody looks. #760 trimmed five blocks of 59 to 100 lines that had grown that way (`_post_rules._mark_suffix_entries`, group's P3 both-sides test, its S2/P2/P4 chain head and S4 chain emitter, and the honorific peel's scope in `_script_segment`), moving into dated decisions bullets what those entries lacked. Two of the moved claims no longer held: a policy example that had stopped reaching its case, and a credential said to land as a title that lands as a suffix. **Writing the user docs (docs/*.rst) has its own AGENTS.md too.** `docs/AGENTS.md` carries the style the docs are written in — tables as indexes, subheadings per task, measured claims, recipe doctests — distilled from PR #588. It loads the same way as the docs/design/ one below; if your tool does not do nested discovery, read it before editing a `.rst` file under docs/. @@ -227,6 +227,7 @@ The compiled sets, read by `Lexicon` (mostly as the `_sets` field of the same na - conjunctions — `CONJUNCTIONS` (e.g. "and", "of") used to chain multi-word titles and to join name parts (rules/P3), plus `CONJUNCTIONS_AMBIGUOUS` (#383/#479, rules/P8, `may_be_initial`, refused by the build on anything but a cased single letter), the subset of single cased letters that read as an INITIAL in a name written wholly in one case — a marker set - maiden markers — `MAIDEN_MARKERS` (`birth_marker` readings, e.g. "née", "geb.") routing the following name to `maiden`; with an `addresses = "given"` title, the one kind a phrase may be (`z domu`) - `config/_ko_surnames.py`, private since 3.0 (`config/surnames.py` through 2.x, 2.8.2 warning of the move) and read as `Lexicon.default().surnames` — `KOREAN_SURNAMES`, the census list the 2.0 API splits unspaced hangul on (#271), still written in Python with an import-time `assert_normalized`: a segmentation list, outside the per-word model, like the locale packs' lists. +- the views' kinds — `GENERATION_WORDS`, `CREDENTIAL_WORDS` (a credential written as a word), `HONORIFIC_WORDS` and `PATRONYMIC_PARTICLES` (3.0's 4b and 4c), private: the kinds the other sets do not record, which classify reads to tag a word for the `generation`, `credentials`, `honorifics` and `patronymic` views (rules/R8, rules/R9) - case masks — `CAPITALIZATION_EXCEPTIONS` mapping of case MASKS (e.g. `{'phd': 'PhD'}`), compiled from the `cased` property, at most one per word: each value is its key's own letters and digits recased, laid over the word as written, so `ph.d.` repairs to `Ph.D.` and repair never writes the value's own punctuation into the word (#459); `Lexicon` raises `ValueError` on a value that spells anything else; and `CAPITALIZED_PARTICLES` (the `saint` and `rank` kinds, private), the particles case repair writes with a capital rather than in lowercase, keyed on the kind so a word needs no mask of its own (rules/R4, #1069) `config/__init__.py` holds a docstring only: the package is the home of the data modules, and since 3.0 nothing else (#357), all of them private (#1007; `regexes` went with #1008; `surnames` became `_ko_surnames` in #1077, named for `vocab/ko.toml`'s tag, and stays in `config/` because the default lexicon reads it and `_lexicon` may not import `locales/`, a layer above it). @@ -236,7 +237,7 @@ The compiled sets, read by `Lexicon` (mostly as the `_sets` field of the same na The 2.0 rewrite landed as underscore-private modules beside the v1 code, which 2.0 and then 3.0 deleted. These conventions apply to all of it. The full design record (rationale, settled-decision logs, dated amendments) lives in untracked `docs/superpowers/specs/`; this section is the enforceable subset. **A commit that establishes or amends one of these conventions must update this section in the same commit** — grep-driven staleness sweeps miss paraphrased prose (see Workflow above), so write-time maintenance is the mechanism, audits are the backstop. - **Module layout**: every module is underscore-private (`_types.py`, `_lexicon.py`, `_policy.py`, `_locale.py`, `_render.py`, `_reading_index.py`, `_pipeline/`, `_parser.py`). The public import surface is exactly `nameparser` and `nameparser.locales` (the vocabulary's compiled sets are private since 3.0, #1007, the Korean surname list since #1077 (2.8.2 warning of it, #1076), and `regexes` went with #1008; `tests/v2/test_layering.py` refuses a public module in `nameparser/config/`); `nameparser/__init__.py` holds re-exports and `__all__` only — no logic. -- **Layering is enforced by `tests/v2/test_layering.py`** (exact-module matching; `if TYPE_CHECKING:` imports don't count): `_types` imports nothing internal at module level — its rendering delegates import `_render` at call time; `_lexicon` and `_policy` sit above `_types` independently (`_lexicon` may import `nameparser.config.*` DATA modules, e.g. `config/_ko_surnames.py` and `config/_compiled.py`, and `_reading_index` at call time only, to compile a configured lexicon's readings, since `tools/vocab/build.py` loads it by path, 3.0's 4i-b); `_reading_index` imports nothing internal at module level and the index, `config/_index.py`, at call time, since `tools/vocab/build.py` loads it by path; `_locale` sits on `_lexicon`+`_policy` (plus `_types` for the shared pickle mixin, and `config/_rules.py` at call time, in `rules_patch`); `_render` imports `_types`, `_lexicon` (for `Lexicon.default()` and `_normalize`) and `_policy` (for the scripts that write no initials); `_pipeline/*` imports `_types`+`_lexicon`+`_policy` plus in-package `_pipeline` helpers, `_assemble` also `_reading_index`, for the kinds `Reading.languages` matches (`CATEGORY`; the files themselves it reads through `Lexicon._files`, 4i-d; `_vocab` imported it in 3.0's 4b and 4c for a listed suffix word's kind and a patronymic particle's, which since 4i-b the lexicon compiles into `_sets`); `_parser` sits on everything except `_render`, `_reading_index` at call time only, to validate a choice's tags and move the case masks it moves (the choice itself compiles in `_lexicon._built`); `__main__` imports only the public package; locale pack modules (`locales/*.py`) import `_locale`/`_lexicon`/`_policy`/`_types` only (`_types` joined the list in #272 — `locales/ja.py`'s segmenter factory constructs a `Segmentation`), and the `locales/__init__` additionally lazy-imports its packs (PEP 562). Extend the test's `ALLOWED` table when adding a module. +- **Layering is enforced by `tests/v2/test_layering.py`** (exact-module matching; `if TYPE_CHECKING:` imports don't count): `_types` imports nothing internal at module level — its rendering delegates import `_render`, and `ParsedName.matches` `_parser`, at call time; `_lexicon` and `_policy` sit above `_types` independently (`_lexicon` may import `nameparser.config.*` DATA modules, e.g. `config/_ko_surnames.py` and `config/_compiled.py`, and `_reading_index` at call time only, to compile a configured lexicon's readings, since `tools/vocab/build.py` loads it by path, 3.0's 4i-b); `_reading_index` imports nothing internal at module level and the index, `config/_index.py`, at call time, since `tools/vocab/build.py` loads it by path; `_locale` sits on `_lexicon`+`_policy` (plus `_types` for the shared pickle mixin, and `config/_rules.py` at call time, in `rules_patch`); `_render` imports `_types`, `_lexicon` (for `Lexicon.default()` and `_normalize`) and `_policy` (for the scripts that write no initials); `_pipeline/*` imports `_types`+`_lexicon`+`_policy` plus in-package `_pipeline` helpers, `_assemble` also `_reading_index`, for the kinds `Reading.languages` matches (`CATEGORY`; the files themselves it reads through `Lexicon._files`, 4i-d; `_vocab` imported it in 3.0's 4b and 4c for a listed suffix word's kind and a patronymic particle's, which since 4i-b the lexicon compiles into `_sets`); `_parser` sits on everything except `_render`, `_reading_index` at call time only, to validate a choice's tags and move the case masks it moves (the choice itself compiles in `_lexicon._built`); `__main__` imports only the public package; locale pack modules (`locales/*.py`) import `_locale`/`_lexicon`/`_policy`/`_types` only (`_types` joined the list in #272 — `locales/ja.py`'s segmenter factory constructs a `Segmentation`), and the `locales/__init__` additionally lazy-imports its packs (PEP 562). Extend the test's `ALLOWED` table when adding a module. - **Canonical field order** — the seven roles in `Role` enum declaration order, defined once and derived everywhere (properties, `as_dict`, reprs, `comparison_key`). Never restate the order literally. `Role` is a `StrEnum`: members compare as their field-name strings, and `tokens_for()` coerces strings via `_coerce_enum`. - **Method organization**, fixed section order in every class: fields + `__post_init__` validation → alternative constructors → dunders (construction/equality → protocol → operators) → properties → public methods by concern (access → editing → comparison → rendering delegates) → private helpers last, except a helper serving exactly one section may sit at that section's head. - **Validation is eager and fail-loud**: every `raise` states the offending value, the expected form, and the fix. Exception taxonomy: wrong type — including wrong element type inside a collection, bare `str` where an iterable of strings is expected, or a `Mapping` where a plain iterable is expected — raises `TypeError`; well-typed but unacceptable values raise `ValueError`; failed enum lookups stay `ValueError` for any input (stdlib `EnumType` precedent). **When the message hands the reader code to paste, that code has to survive a type checker, and then RUN** — nameparser ships `py.typed`, and a placeholder can type-check and still raise: the 2.4 release sweep's first fix to the `capitalization_exceptions` hint offered `('key', 'Value')`, which passes mypy and fails #459's mask check, so `test_add_capitalization_exceptions_raises_pointing_at_replace` now executes the offered code. #337's segmenterless warning offered `Policy(segment_scripts=())`, an `arg-type` error, because these fields are annotated with what they STORE rather than everything the constructor accepts. Prefer the `frozenset()` / `()` spellings in messages and docstrings, and pin the offered spelling in a test — the warning tests matched on `ja_segmenter` and never checked the actionable half of the message. **A warning emitted in `Parser.__post_init__` needs `parser_for` to re-emit it from its own frame** (the `catch_warnings(record=True)` block at its return): `__post_init__`'s `stacklevel` is sized for direct `Parser(...)` construction, and through `parser_for`'s extra frame the default one-line rendering attributes the warning to the library's own `return Parser(...)` — the exact call the message tells the user to change becomes invisible. No single stacklevel serves both entry points; a new construction warning gets the re-emission for free, but a new CONSTRUCTION SITE for `Parser` inside this package needs its own re-emission or its callers get library-attributed warnings (#337 review). @@ -252,8 +253,8 @@ The 2.0 rewrite landed as underscore-private modules beside the v1 code, which 2 - **A claim about WHICH STAGE or WHICH LAYER does something is checkable — check it before writing it.** The pipeline is eight stages with a written ownership map (`ParseState`'s docstring, pinned by `tests/v2/pipeline/test_state.py`), and `parse(s).tokens` prints every token's role and tags, so "extract assigns this", "classify never sees that", "group consumes it" each have a one-command answer. #329's prose claimed delimited maiden content is *"claimed whole before classify has tagged anything inside it"*; measured, `classify` tags the marker fine and only the CONSUMING is missing, because `_group`'s rule walks `pieces` and a token that already carries a role is not in `pieces`. Two different mechanisms, one plausible sentence covering both. That single claim then shipped SIX times across three correction rounds, which is the part worth internalizing: **when a mechanism claim turns out wrong, sweep for where else you wrote it, and sweep again at the END of the change over the words the change itself just added.** Prose density here means one idea lives in a docstring, a case note, a release-log entry and this file at once; the implementer working against a wrong mechanism is the person most likely to restate it; and rewriting a mechanism claim is writing one, so the correction earns the same one-command check as the original — two of the six instances were fresh errors introduced by the sentence fixing the previous one. **Adjacency is the trap.** The claim that feels already-known is the one about the neighbouring stage or the neighbouring layer: one comment block in the 2.x `tests/v2/test_facade_cases.py` got the exception type, the raising layer, the skip mechanism, the count of skipped rows, and which row was blocked all wrong at once — every one a claim about `_config_shim` versus real 1.4.0, written from reasoning, in the file whose whole job is translating between them. What finally held was not better prose but moving the claim into a test (`_CORE_ONLY_IDS`, deleted with the facade in 3.0), which cannot be wrong the way a sentence can. - **Typing/docs**: `from __future__ import annotations`; `frozen=True, slots=True` on every public dataclass; strict-profile mypy flags via per-module overrides in pyproject (`strict = true` itself is not valid per-module). Docstrings state contracts in prose with **no doctest blocks** — `--doctest-modules` makes every example a test; behavior examples go to unit tests per the lean-docs rule. **Document the positive direction of a partial property**: "a non-empty `ambiguities` is a signal to act on" is checkable, while "an empty one means no fork occurred" is a universal negative needing exhaustive verification -- that claim was written twice and falsified twice, at sites the author had not audited. - **The segmenter contract**: the optional `Parser(segmenter=...)` hook is parse-totality's ONE exception (locales spec section 4). Everything inside that exception is a bug in USER CODE, never a fact about the name, so it is surfaced rather than absorbed: the segmenter's own exceptions propagate, and the two protocol violations the stage can detect for itself — an answer of the wrong type, and one cutting at or past the end of the token it was handed — raise `TypeError`/`ValueError` from `_script_segment` for the same reason. The line to hold when adding a check there: a protocol violation by the segmenter's AUTHOR raises, while an adapter's defense against its own third-party library (`locales/ja.py`'s repertoire, length, reconstruction and score guards) declines with `None`, because what those catch is a fact about the content. -- **Pickling**: v2 types must round-trip (`Parser` is picklable by construction, and it holds a `Lexicon`; the one qualifier is that a `Parser` pickles iff its segmenter does — see the segmenter bullet above). Every frozen type assigns `_guarded_getstate`/`_guarded_setstate` (`_types.py`) in its class body (`@dataclass(slots=True)` would override inherited pickle methods) — unpickling fails at the LOAD site on field-layout skew, with one widening: a field added after its type shipped carries `metadata={_types.ADDED_IN: ""}` and a default, and an older pickle missing it loads with the default (`Ambiguity.alternative`, 2.7; `Token.reading`, `Reading.languages`, `Reading.second_surname`, `Reading.first_surname` and `ParsedName.order`, 3.0.0) -- a missing field without the marker, default or not, an unknown field and a renamed one are still refused, each a row in `tests/v2/test_types.py::_REFUSED_SKEWS`, so a new field on a pickled type takes the marker or breaks every older pickle; the shared guard deliberately does NOT re-validate values (pickle is not a security boundary; canonical state comes from a validated instance). `Lexicon` and `LexiconReading` keep their own copy of the guard (layering); a new unpicklable slot type needs the same treatment plus a round-trip test. Since 3.0's 4i-b a `Lexicon` pickles its lists, its masks, its edit log, its choice of languages (`_languages` and `_strict`, 4i-c, marked `ADDED_IN` so a 4i-b pickle loads choosing nothing, a 2.x state translating to no choice) and its base as a SNAPSHOT of the reading index's rows, so a pickle parses as it did where it was written, a snapshot equal to the running release's index loading as the by-reference default again (decisions/vocabulary-model, 2026-11-02 #754, item 5); its private `_sets`, `_cap_map`, `_chosen` and `_files` fields are built at construction and never pickled. `Lexicon`'s copy of the guard admits the `ADDED_IN` marker, read under `_lexicon._ADDED_IN`, the same string, since layering keeps `_lexicon` from importing `_types` (`test_lexicon_marks_an_added_field_with_the_shared_key` holds the two equal), so a new `Lexicon` field takes the marker and a default or breaks older pickles. A 2.x state holds the compiled sets instead, and `_lexicon._translated_2x` translates it onto the nearer named base and a per-word log, loading where the compile gives the sets back and refusing otherwise, naming the fields, so the 2.8 `Lexicon`, `Parser` and locale-pack pickles in `tests/v2/data/lexicon_v28.pickle` load as the 3.0 calls that build them, each `Lexicon` loaded warning with a `DeprecationWarning`: the translation is migration scaffolding, deleted in 3.1 (decisions/vocabulary-model, 2026-11-02 #754, the 2.x translation's lifetime), so build nothing new on it. And unlike the shared guard it DOES re-validate: a load goes through the constructor, which checks the base, refuses an edit not in canonical form, and refuses a list or mask entry that is not already normalized, because an old blob would otherwise load a different lexicon in silence (`particles_ambiguous` changed meaning between v1 and v2 without changing its name). -- **One sanctioned global**: the (future) cached default `Parser`. Lazily cached FROZEN singletons (`Lexicon.default()`'s `functools.cache`, the future default parser) are constants, not state; any second piece of module-level MUTABLE state requires amending the conventions doc, on purpose, in review. +- **Pickling**: v2 types must round-trip (`Parser` is picklable by construction, and it holds a `Lexicon`; the one qualifier is that a `Parser` pickles iff its segmenter does — see the segmenter bullet above). Every frozen type assigns `_guarded_getstate`/`_guarded_setstate` (`_types.py`) in its class body (`@dataclass(slots=True)` would override inherited pickle methods) — unpickling fails at the LOAD site on field-layout skew, with one widening: a field added after its type shipped carries `metadata={_types.ADDED_IN: ""}` and a default, and an older pickle missing it loads with the default (`Ambiguity.alternative`, 2.7; `Token.reading`, `Reading.languages`, `Reading.second_surname`, `Reading.first_surname`, `ParsedName.order`, `Policy.two_given_names`, `PolicyPatch.two_given_names` and `Locale.languages`, 3.0.0) -- a missing field without the marker, default or not, an unknown field and a renamed one are still refused, each a row in `tests/v2/test_types.py::_REFUSED_SKEWS`, so a new field on a pickled type takes the marker or breaks every older pickle; the shared guard deliberately does NOT re-validate values (pickle is not a security boundary; canonical state comes from a validated instance). `Lexicon` and `LexiconReading` keep their own copy of the guard (layering); a new unpicklable slot type needs the same treatment plus a round-trip test. Since 3.0's 4i-b a `Lexicon` pickles its lists, its masks, its edit log, its choice of languages (`_languages` and `_strict`, 4i-c, marked `ADDED_IN` so a 4i-b pickle loads choosing nothing, a 2.x state translating to no choice) and its base as a SNAPSHOT of the reading index's rows, so a pickle parses as it did where it was written, a snapshot equal to the running release's index loading as the by-reference default again (decisions/vocabulary-model, 2026-11-02 #754, item 5); its private `_sets`, `_cap_map`, `_chosen` and `_files` fields are built at construction and never pickled. `Lexicon`'s copy of the guard admits the `ADDED_IN` marker, read under `_lexicon._ADDED_IN`, the same string, since layering keeps `_lexicon` from importing `_types` (`test_lexicon_marks_an_added_field_with_the_shared_key` holds the two equal), so a new `Lexicon` field takes the marker and a default or breaks older pickles. A 2.x state holds the compiled sets instead, and `_lexicon._translated_2x` translates it onto the nearer named base and a per-word log, loading where the compile gives the sets back and refusing otherwise, naming the fields, so the 2.8 `Lexicon`, `Parser` and locale-pack pickles in `tests/v2/data/lexicon_v28.pickle` load as the 3.0 calls that build them, each `Lexicon` loaded warning with a `DeprecationWarning`: the translation is migration scaffolding, deleted in 3.1 (decisions/vocabulary-model, 2026-11-02 #754, the 2.x translation's lifetime), so build nothing new on it. And unlike the shared guard it DOES re-validate: a load goes through the constructor, which checks the base, refuses an edit not in canonical form, and refuses a list or mask entry that is not already normalized, because an old blob would otherwise load a different lexicon in silence (`particles_ambiguous` changed meaning between v1 and v2 without changing its name). +- **One sanctioned global**: the cached default `Parser` (`_parser._default_parser`, which `parse()` and `ParsedName.matches` use). Lazily cached FROZEN singletons (`Lexicon.default()`'s `functools.cache`, the default parser's) are constants, not state; any second piece of module-level MUTABLE state requires amending the conventions doc, on purpose, in review. - **Tests**: all v2 tests live in the `tests/v2/` package, one test module per source module plus the cross-cutting ones (`test_reprs.py`, `test_layering.py`, `test_contracts.py`, `test_properties.py`, `test_benchmark.py`, the `cases.py`/`test_cases.py` table, `test_regex_sync.py`, which pins render's hand-copied patterns to the pipeline's, and `test_snapshot.py`, which holds every corpus name's reading to `tools/differential/snapshot.jsonl` — sharing `_differential_fixtures.py` with `test_differential.py` and `test_corpora.py` for the handles on `tools/differential/`), names stating behavior; the 1.x suite's parsing tests stand ported to `parse()` in `tests/*.py`. Never assert `Lexicon.default()` contents; the narrow sourcing spot-checks in `test_default_sources_v1_vocabulary` that pin the v1→v2 migration contract (e.g. the flipped `particles_ambiguous` model) are the sanctioned exception. ## Extension Patterns @@ -278,13 +279,13 @@ The 2.0 rewrite landed as underscore-private modules beside the v1 code, which 2 **Parsing must never write into its config** — `Lexicon` and `Policy` are frozen, so a parse cannot write to them; v1 did, registering dotted abbreviations like `Lt.Gov.` into the shared config through 1.2.x, which made parse results order-dependent and thread-unsafe. Keep any new cache keyed and immutable for the same reason (`tests/test_constants.py`'s `JoinedPieceTests` hold the readings of that bug's inputs). -**The `nameparser/config` vocabulary constants are frozen** (2.2, #293) — `TITLES.add("dean")` raises `AttributeError` at the line that writes it, so a runtime addition goes on a config OBJECT instead: `Parser(lexicon=Lexicon.default().remove("dean", kind="given"))`. What the freeze buys: `Lexicon.default()` is `functools.cache`d and reads the constants once, so pre-freeze an edit after the first parse was invisible to `parse()` while one before it was not, which depended on nothing the reader could see (through 2.x the v1 `Constants` copied the constants at every construction too, a third moment). +**The `nameparser/config` vocabulary constants are frozen** (2.2, #293; private since 3.0, #1007) — `_compiled.TITLES.add("dean")` raises `AttributeError` at the line that writes it, so a runtime addition goes on a config OBJECT instead: `Parser(lexicon=Lexicon.default().remove("dean", kind="given"))`. What the freeze buys: `Lexicon.default()` is `functools.cache`d and reads the constants once, so pre-freeze an edit after the first parse was invisible to `parse()` while one before it was not, which depended on nothing the reader could see (through 2.x the v1 `Constants` copied the constants at every construction too, a third moment). **Titles permanently shadow first names — be conservative** — any word in `TITLES` is always consumed as a title and can never be parsed as a first name. `"Dean"` is the canonical example: it's a common academic title *and* a common given name, so it is not in `TITLES` but in `TITLES_AMBIGUOUS`, a title only inside a title phrase (`Dean of Students`, `Associate Dean`; rules/H9, #752 phase 2i), and users who need it everywhere add it to a `Lexicon` of their own. A title that is also a given name now has that machinery: a `given` reading on the word moves it there, decided word by word in measured batches (decisions/vocabulary-model's 2026-10-24 order). A rule that reads a title by its neighbours in the title run meets more phrase shapes than the qualifier-then-head it was designed on: the object of a connective (`President of the United States`, `the Queen's Bench`, whose `united`, `states` and `bench` are modifiers with no head after them), a head before `of` (`Member of Parliament`), and a qualifier written after its head (`Chargé d'Affaires`, `Princess Royal`). Phase 2i's first build, "a modifier needs a head title after it", broke `President of the United States`, `Chargé d'Affaires` and the committed `Queen's Bench` row, found by the case table and the committed corpora, and then read a modifier ENDING the name out of H5's trailing run (`Jane Smith Vice.`), found by the design-docs review; put each shape, and the trailing position, in the grid of any such rule. Before adding a word to `TITLES`, ask: "Could this plausibly be someone's given name in any culture?" If yes, don't add it globally; it belongs in a caller-supplied `Lexicon` instead. This same caution applies to international honorifics — `Frau` is a first name in some contexts, and `Prince` and `Sheikh` were titles until 2.7 gave them a `given` reading, which moved them to `TITLES_AMBIGUOUS` (rules/H9). It also applies to any prefix sub-set gated on "never a first name": obscure-looking foreign particles are surprisingly often real given names — `Von` (Von Miller), `Vander` (Brazilian, also the Arcane character). When unsure, exclude — a missing member just means that name isn't auto-handled, whereas a wrong member misparses a real person. The decided exception is a word borne as a name only RARELY in the leading position — mostly a surname written last (`graf`, `greve`, `knight`), a few with small given-name or surname-first counts (`herra`, `hrabia`): the title claim acts only in front, so it ships, and a name that begins with it (`Greve Anna`, `Herra Wijaya` → title) is the accepted cost. That is a judgment by incidence, made per word, with the counts taken recorded on the word's entry in `nameparser/vocab/` (herra's, hrabia's; graf, which predates #606, and greve and knight, which #606 proposed, carry none) and the class in `decisions/salutation-titles` (rules/H1's Accepted clause states the cost), not a rule a sweep can apply: a word that commonly leads a real name (Vietnamese `Ông`, the given names `thiru`, `puan`) is still out of `TITLES`; `marshal` reads as a title only in a phrase (`Field Marshal`). `tools/name_usage.py` produces the counts this judgment starts from (role counts, plus Wikidata citizenship to tell where a family name is written first); its Wikidata column is a lower bound (Shehu Shagari has no given-name statement), so a small count there proves little and a zero proves nothing, and its nam_dict row gives a band per country rather than a count, with a missing name proving nothing either. **The period-abbreviation title inference runs at the head of the GIVEN-NAME part, not the head of the name** — an unrecognized multi-letter word ending in a single trailing period (`_pieces._PERIOD_ABBREV`, v1's `period_abbreviation` regex, imported from `_vocab`, `{2,}` letters — it was assign's until #424 and group's until #439) is treated as a title in the leading title run, e.g. `"Insp. Jane Morse"` → `title='Insp.'`. "Leading" is per SEGMENT: `_peel_leading_titles` is called for NO_COMMA segment 0, SUFFIX_COMMA segment 0, and FAMILY_COMMA **segment 1**, so `"Morse, Det. Insp. Jane"` → `title='Det. Insp.'` and a lone `"Smith, Xyz."` → `title='Xyz.'` — long-standing, verified against 1.4.0, and the mechanism behind #296 (`"Smith, Jr."` read title in 1.4.0; the comma's credential read takes it as suffix `Jr.` now). The docs said "leading word" until 2026-08-01 and were wrong for every comma path. FAMILY_COMMA **segment 0** peels too since #701, but by VOCABULARY only (rules/H6, in assign's family-comma branch rather than `_peel_leading_titles`): `Dr. Smith, John` → `title='Dr.'`, while `Xyz. Smith, John` keeps family `Xyz. Smith`, and a title-particle (`St`, `Freiherr`) stops the run, as does a title with nothing but post-nominals behind it (`Dr. Jr., John` keeps family `Dr.`); a title/suffix dual with a name word behind it still peels (`Rev. Sr. Smith, Mary`). It adds nothing to the `Lexicon`, so the periodless form (`"Insp"`) is unaffected elsewhere. The `{2,}` length requirement — not a separate initials check — is what excludes single-letter initials like `"J."`; the same word after the given name is left as a middle name. **The inference OUTRANKS vocabulary where it runs**: `"Esq. Smith"` → `title='Esq.'` even though `esq` is suffix-only vocabulary, because the shape rule fires before anything consults the suffix sets. The one vocabulary it yields to, since #789 (phase 2k of #752), is a given name written short (a `given` reading with `same_as`, in the TOML or a caller's `add(w, kind="given", same_as=...)`, compiled into the internal `Lexicon._sets.given_name_abbreviations`): `"Chas. Smith"` → `given='Chas.'`, read off classify's `vocab:given-abbreviation` tag in `is_leading_title` and off the lexicon in `_vocab.is_title_shaped`, the two spellings of one test that `test_is_title_shaped_and_is_leading_title_agree` holds together. The second spelling is what rules/C4's comma count and the title phrase's head run ask, so a change to the shape moves comma names nobody wrote it for: #789's first grid held no comma and missed `Chas. Smith, Ed` (given 'Ed', family 'Chas. Smith' before, suffix 'Ed' after), which only mutating `is_title_shaped`'s half alone showed. Put a word of the ambiguous class after a comma behind the shaped word (`X. Smith, Ed`), and a modifier in front of it (`Vice X. Smith, Ed`), in the grid of any change to the shape, and mutate each spelling separately. **The INFERENCE still runs in one direction only, and that is what the two slots share and where they part** (rewritten 2026-09-08, #316): a period-marked word is claimed by SHAPE at the front and by VOCABULARY at the back, so an unlisted abbreviation opening a name is a title while an unlisted abbreviation ending one is a NAME word (`"John Smith Xyz."` → `family='Xyz.'`). A trailing period-marked word the vocabulary knows as a title now IS one (`"John Smith Prof."` → `title='Prof.'`, `rules/H5`) — the comma path read a lone `Smith, Prof.` that way already, and the given part's `Smith, John Prof.` since 2.3.0 (1.4.0 through 2.2.0 read `Doe, Jane Mrs.` middle 'Mrs.'), so the two agree now — and a trailing period-marked word the UNAMBIGUOUS SUFFIX vocabulary knows is still a suffix, the suffix run being peeled first (`"John Smith Esq."` → `suffix='Esq.'`). A BARE trailing title word stays a name word (`"John Smith Sir"`, `"Mary Jane Pope"`), which is the doctrine line below, save a listed salutation ending the given part after a family comma (rules/H8, `"Doe, Jane Mrs"` → `title='Mrs'`). Meanwhile `period_joined_vocab` resolves INTERIOR-period tokens (`Lt.Gov.`, `Msc.Ed.`) to title-or-suffix by vocabulary, and `_extract._period_escape` treats any period-final delimited content that rules/S1 does not set apart as not-a-nickname. Four sites, and the trailing one now resolves by two vocabularies in order rather than one; unifying the rest is open design work, not settled. (#109; see `docs/usage.rst` "Titles you didn't configure") -**Cyrillic suffix regexes need `re.I` even when the pattern is suffix-only** — a Latin title-cased word (`Ivanovich`) keeps its suffix lowercase, so `re.I` seemed skippable; but an irregular Cyrillic suffix can be nearly the whole word (`ильич`), so title-casing capitalizes into the suffix itself (`Ильич`). `east_slavic_patronymic_cyrillic` shipped without `re.I` on the Latin reasoning and silently failed on capitalized irregular forms — don't assume Latin's title-case safety transfers to Cyrillic. (#185) +**Cyrillic suffix regexes need `re.I` even when the pattern is suffix-only** — a Latin title-cased word (`Ivanovich`) keeps its suffix lowercase, so `re.I` seemed skippable; but an irregular Cyrillic suffix can be nearly the whole word (`ильич`), so title-casing capitalizes into the suffix itself (`Ильич`). `east_slavic_patronymic_cyrillic` (v1's name for what is `_vocab._EAST_SLAVIC_CYR` since 3.0 removed `config/regexes.py`, #1008) shipped without `re.I` on the Latin reasoning and silently failed on capitalized irregular forms — don't assume Latin's title-case safety transfers to Cyrillic. (#185) **Suffix words vs the initial veto** — single-letter roman numeral suffixes (`i`, `v`) are suffix words but also initial-shaped, so `_vocab.is_suffix_strict` (v1's `is_suffix`) vetoes them through `_vocab.is_initial`. `_vocab.is_suffix_lenient` accepts suffix words without the veto and is only safe in unambiguous positions, after a comma (#136, #144). **The veto is Latin-only since #320**: `is_initial` ANDs the bare shape test (`is_initial_shaped`) with "not in `_policy._NO_INITIALS`", the four scripts that have no initials (Han, Hangul, Hiragana, Katakana), so the single-character CJK suffix words (씨, 様, 氏, 군, 양, 님, 殿) written with a period read as suffixes: `"김민준 씨."` gives suffix `씨.`. `V.`/`I.` still lose to the veto and still need the lenient test. @@ -300,7 +301,7 @@ The 2.0 rewrite landed as underscore-private modules beside the v1 code, which 2 **`_normalize` must reach a fixed point** — storage and match-time share the one fold, and `Lexicon.__setstate__` re-validates, so a value that changes on re-normalization changes under its owner. `strip().strip(".")` alone is not idempotent (`'. a .'` → `' a '` → `'a'`). The loop is the fix; keep any new stripping inside it. **Anything built on `_normalize` must converge too** — `_fold_words` runs `_normalize` per word and DROPS the words that fold away (`_title_key` is that list space-joined, and `_run_addresses_by_given` reads the list itself, so its last-word arm is the last word of the FOLDED key by construction); keeping the empty slot stored `'lt .'` as `'lt '`, a key match-time can never rebuild (so the entry is silently inert) and `__setstate__` rejects on the next round-trip as "not written by this version". -**Perf regressions are caught by the scaling tests, not the absolute-time ones** — the `_thousand_names` tests use constant-size, delimiter-free input and are structurally blind to a complexity regression. A new inner loop, a policy-gated stage or a shape that needs a prefix gets a row in `tests/v2/test_benchmark.py`: its module docstring says which table, and the comment beside each table carries the measurements behind its rule. Coverage is collected on ONE interpreter, 3.14's (`COVERAGE_ON` in `.github/workflows/python-package.yml`, whose comment says why). +**Perf regressions are caught by the scaling tests, not the absolute-time ones** — the thousand-names test (`test_a_thousand_names_still_parse_in_reasonable_time`) uses constant-size, delimiter-free input and is structurally blind to a complexity regression. A new inner loop, a policy-gated stage or a shape that needs a prefix gets a row in `tests/v2/test_benchmark.py`: its module docstring says which table, and the comment beside each table carries the measurements behind its rule. Coverage is collected on ONE interpreter, 3.14's (`COVERAGE_ON` in `.github/workflows/python-package.yml`, whose comment says why). **A JSONL file written with `json.dumps(..., ensure_ascii=False)` is cut by `str.splitlines()`** — `json.dumps` leaves U+2028, U+2029 and U+0085 unescaped, and `splitlines()` splits at all three, so one name pasted from a document holding a line separator turns its line into two halves of invalid JSON, and the file into one no reader can load. #801's review found it in the local corpora (a single official among 536,981 would have made the cache unloadable for good), and the review of that fix found it again one file over, in `snapshot.py`'s `dump`/`load`, which the fix had not touched. So it is fixed at the shared source now: `corpus_build.jsonl_line` escapes the three and `corpus_build.jsonl_lines` cuts at `"\n"` only, and `corpus_build`'s corpus and fields writers, `snapshot.py`, `cldr_agreement.py`, `local_corpora.py`, `local_psc.py`'s extracts and `local_agreement.py` go through them. A new JSONL writer or reader in tools/ goes through the same two. Still outside them: the builders that write their own lines (`build_cjk_corpus.py`, `_conventions`, `_shapes`, `_rules`, `_issues`), `compare.py`'s worker stream, the tests' corpus reads (`_differential_fixtures.py`, `test_corpora.py`), and the extracts of `local_orcid` and `local_patentsview`, streamed through `gzip.open(..., newline="\n")` rather than read whole for their 25M and 5.3M lines, which cuts at `"\n"` only as `jsonl_lines` does; none of the 16 committed JSONL files held one on 2026-10-21. diff --git a/README.rst b/README.rst index 988dde7f1..1e667a463 100644 --- a/README.rst +++ b/README.rst @@ -13,7 +13,10 @@ compatibility layer.** ``HumanName``, ``Constants``, the shared names of the word lists; ``parse()`` and ``Parser`` are the replacement. 2.8 is the last release that has them, and it warns at each use 3.0 removes; `Coming from 1.x or 2.x`_ below says where to -start. The `release log +start. 3.0 also configures a ``Lexicon`` by reading +(``Lexicon.default().add("leed", kind="credential")``), renders a name +in the order it was written (``str(parse("김민준"))`` is ``'김민준'``), +and adds packs that choose the languages a parse reads. The `release log `__ has the details. Please `open an issue `__ for anything @@ -98,11 +101,11 @@ Coming from 1.x or 2.x 3.0 has no ``HumanName``: ``parse(text)`` reads the same seven fields, named ``given`` and ``family`` where ``HumanName`` had ``first`` and ``last``, and a ``Lexicon`` and ``Policy`` handed to a ``Parser`` -replace ``Constants``. Upgrade to 2.8 first and run your tests under +replace ``Constants``. Upgrade to 2.8.2 first and run your tests under ``python -W error::DeprecationWarning``: it stops at every use 3.0 -removes except a read of the shared ``CONSTANTS`` and the imports -`Migrating to 3.0`_ lists, which 2.8 does not warn on, so search for -those too. A pickled ``HumanName`` or +removes except a read of the shared ``CONSTANTS`` or of a ``Lexicon`` +set field and the imports `Migrating to 3.0`_ lists, which 2.8.2 does +not warn on, so search for those too. A pickled ``HumanName`` or ``Constants`` does not load in 3.0; store ``parse(text).as_dict()`` or the original string instead. `Migrating to 3.0`_ maps ``HumanName``'s attributes and the diff --git a/docs/AGENTS.md b/docs/AGENTS.md index 97b00898c..ba098b9be 100644 --- a/docs/AGENTS.md +++ b/docs/AGENTS.md @@ -22,7 +22,7 @@ The rules below were distilled from PR #588, which restructured `customize.rst` **A recipe is a doctest; an edge case is a unit test.** A *recipe* — how to do the thing the reader came to do — belongs in the docs as a `.. doctest::` block, which teaches and is CI-checked at once. The line between the two is whether the prose states the behavior: a claim the section makes to the reader (`edd` repairs to `EDD`, not a mask) earns a doctest beside it, which is what keeps the claim true; an edge case the prose has no reason to mention earns one sentence at most and a unit test, never a doctest (PR #216). An example sitting as prose inside a table cell is the worst of both: it teaches from the wrong place and nothing checks it. -**An example must be non-vacuous.** Show the baseline before the customization, and check that they differ: an `add(titles={"chancellor"})` example once shipped (fixed in 0ab19fdb) where `chancellor` was already a default title, so the add was a no-op and the doctest passed demonstrating nothing. Prefer a word absent from the defaults for a teachable reason (`dean`, which is also a given name). +**An example must be non-vacuous.** Show the baseline before the customization, and check that they differ: an `add(titles={"chancellor"})` example (2.x's spelling) once shipped (fixed in 0ab19fdb) where `chancellor` was already a default title, so the add was a no-op and the doctest passed demonstrating nothing. Prefer a word absent from the defaults for a teachable reason (`dean`, which is also a given name). **An example must run warning-free.** Readers paste examples, so a warning the doctest build prints is one every reader gets. Fix the example, not the build: a `Lexicon.empty()` used only to drop titles also dropped the Korean surnames and drew the segmenterless warning, and the fix was an example that empties only the field it is about. Run the doctest build under Verifying below and expect no output at all. @@ -44,7 +44,7 @@ The rules below were distilled from PR #588, which restructured `customize.rst` ## Linking -**Link, don't restate.** When another page already covers a topic, give one line and a `:ref:` to it; two copies drift apart. When the content is a list the code ships (the maiden markers, the delimiter pairs), link the module with `:mod:` instead of copying the list into prose, where it goes stale the next time the list changes. +**Link, don't restate.** When another page already covers a topic, give one line and a `:ref:` to it; two copies drift apart. When the content is a list the code ships (the maiden markers, the delimiter pairs), link the object that holds it (`` :data:`~nameparser.DEFAULT_NICKNAME_DELIMITERS` ``) or show the call that lists it (`Lexicon.default().words(kind="birth_marker")`) instead of copying the list into prose, where it goes stale the next time the list changes. Since 3.0 no vocabulary module is public for `:mod:` to name (#1007). **Put a label at the heading you link to**, and link with `:ref:` to the section rather than `:doc:` to the page, which lands the reader at the top of a long page. Add the label in the same commit as the first link to it. diff --git a/docs/concepts.rst b/docs/concepts.rst index db6812c06..a67994c2d 100644 --- a/docs/concepts.rst +++ b/docs/concepts.rst @@ -184,7 +184,7 @@ threads without locking, safe to use as dict keys or cache keys, and equality means exactly what it says — two :class:`~nameparser.Parser` instances built from equal configuration are equal values, not merely two objects that happen to behave the same. Every piece of -configuration in the 2.0 API is a frozen value — including the +configuration is a frozen value — including the module-level default parser itself. Honest ambiguity diff --git a/docs/customize.rst b/docs/customize.rst index 51ec0b2d2..aa5976de3 100644 --- a/docs/customize.rst +++ b/docs/customize.rst @@ -1123,7 +1123,9 @@ rarely surnames tell the two shapes apart, so the switch leaves a word listed in the lexicon's ``given_names`` where it stands, under any ``name_order``. The field is empty by default, so you supply the list for your data, or choose ``locales.ES``, which turns the switch on and -carries 1,000 Spanish given names; an entry written without accents +carries 1,000 Spanish given names (``locales.ES_AR``, for Argentine +names, carries 5,000; see :ref:`spanish-portuguese-titles`); an entry +written without accents matches the word with them: .. doctest:: @@ -1800,7 +1802,10 @@ at the point of output, not baked into the parse. Three methods on signatures: * :meth:`~nameparser.ParsedName.render` fills a format spec from the - seven role fields. + seven role fields and the views derived from them (``family_base``, + ``generation``, ``credentials`` and the rest). With no spec, as + ``str()`` calls it, it lays the name out in the order it was read in + (since 3.0). * :meth:`~nameparser.ParsedName.initials` is the same idea narrowed to first letters, with its own ``delimiter``/``separator`` arguments. * :meth:`~nameparser.ParsedName.capitalized` returns a new, case-fixed diff --git a/docs/design/decisions/3-0-reevaluations.md b/docs/design/decisions/3-0-reevaluations.md index 6b87c58d8..5d9c91093 100644 --- a/docs/design/decisions/3-0-reevaluations.md +++ b/docs/design/decisions/3-0-reevaluations.md @@ -6,7 +6,7 @@ Promoted 2026-08-15 from session memory (Derek's 2026-07-30 ask; promotion appro - 3.0 carries no v1 compatibility layer: `HumanName`, `nameparser.parser`, `nameparser.util`, `nameparser.config`'s five shim names and the five 1.x word-list names 2.2 retired are gone, so every surface speaks `given` and `family`, a configuration is a `Lexicon` and a `Policy` handed to a `Parser`, and a pickle is a core type's — 2026-10-28 #357, REMOVAL. - Render's default no longer space-joins its fields as v1's did: 3.0's default follows the order a name was read in and keeps its written separators (`str(parse("威廉·莎士比亚"))` is `'威廉·莎士比亚'`, where 2.x gave `'威廉 莎士比亚'`), decided in decisions/vocabulary-model's #754 output-model entry, item 6, and stated as rules/R6 — 2026-08-15; 2026-10-28 #754 (Derek), THE RENDER SIGNATURE. -- A `.pyi` stub typing `Policy.__init__` wide and its attributes narrow is a 3.0 candidate, deferred from #334 for want of a signature list with its own sync test — 2026-08-15. +- A `.pyi` stub typing `Policy.__init__` wide and its attributes narrow is unbuilt, a 3.x candidate (3.0 shipped without it), deferred from #334 for want of a signature list with its own sync test — 2026-08-15; 2026-11-05 #1079. - `nameparser.config` holds data modules and nothing else in 3.0 — 2026-10-28 #357 (Derek); the five shim names left it, and the five 1.x word-list names 2.2 retired went with their alias bridge — 2026-10-28 #357, REMOVAL; the modules re-exporting the compiled vocabulary (`titles`, `particles`, `suffixes` and five more) left it too, `Lexicon.words(kind=...)` listing a kind's words, and `regexes` left it (2026-11-02 #1008), both removals warned of by 2.8.1 — 2026-11-02 #1007 (Derek); and `surnames` became the private `_ko_surnames`, so the package holds no public module, `Lexicon.default().surnames` holding its words, warned of by 2.8.2 — 2026-11-04 #1077. - Pickle-guard layout breaks landing in a minor release are shim-era friction; the guarded raise stands on its own — 2026-08-15. - Reading positionally where no script licenses family-first is the safe direction on its own, family-first being the marked case that needs affirmative evidence — 2026-08-15. @@ -24,7 +24,7 @@ Not standing: the harness bullet, that the snapshot and `compare.py` cannot show Elsewhere: the explicit-comma line narrowed to a description for CJK script — decisions/cjk-comma-demotion's 2026-09-01 #469, SCOPES the FAMILY_COMMA bullet entry; the 1.4 gate the harness-baseline bullet expected 3.0 to dissolve, already retired, `compare.py --baseline 1.4.0` answering the question when asked — decisions/differential-ledger's 2026-10-13 #664 entries; the vocabulary model packs are to narrow, a strict pack dropping other languages' readings — decisions/vocabulary-model's 2026-10-15/16 entry; what the shim's filter kept and dropped until #357 — decisions/R4's 2026-10-02 #541 and 2026-10-02 #582 entries. -Open: #753 (phase 3, language packs; the three pack bullets carried out by its 3b, 2026-10-29, the packs themselves to come). +Open: #753 (phase 3, language packs; the three pack bullets carried out by its 3b, 2026-10-29, and 3.0's packs shipped after it, from `locales.NE` to `locales.ES_AR`; a pack past 3.0, as `is` is in #1074, is the issue's). ## Log @@ -90,3 +90,4 @@ Open: #753 (phase 3, language packs; the three pack bullets carried out by its 3 - **Why it does not stay.** The 2026-11-02 entry kept the module because its list is data of its own, outside the per-word model; that says where the list is written, not that it needs a public import path. The words are public already as `Lexicon.default().surnames`, the field 4i-b kept public (decisions/vocabulary-model, 2026-11-02 #754), and the constant is that field's value, equal under 2.8.1 against its wheel (measured 2026-11-04), and 2.8.2's tests/v2/test_removals_2_8_1.py runs the offered code and holds it equal. A second public spelling of one set is one more surface to keep, and the only module left in a package that otherwise holds none. - **Where it lives, and its name.** In `nameparser/config/`, because `Lexicon.default()` reads it and `_lexicon` may not import `nameparser.locales`, the layer above it (tests/v2/test_layering.py); the lists only a pack reads stay beside their packs (`locales/_es_given_names.py`, `_es_ar_given_names.py`). `_ko_` follows the BCP 47 tag naming of `vocab/ko.toml`; the constant keeps its name, `KOREAN_SURNAMES`. - **The warning.** 2.8.1's mechanism unchanged (`config/_deprecated.py`'s `alias_getattr` with `removed=`): on 2.x the list moved to `_ko_surnames.py`, which the library reads, and `surnames.py` re-exports it under the hook, so only a caller's read warns, naming `Lexicon.default().surnames`, which runs quietly on 2.8.2 and 3.0 alike. The message cites #1007, so pyproject's filter for that family covers it. `tests/v2/test_layering.py` now refuses a public module in `nameparser/config/`. +- 2026-11-05 #1079 — THE `.pyi` STUB FOR `Policy` IS NOT BUILT FOR 3.0 AND STAYS A 3.x CANDIDATE. The 2026-08-15 bundle and decisions/differential-ledger's 2026-08-05 #334 deferral named it a 3.0 candidate; 3.0 ships without it, decided in chat during the 3.0 release sweep. It needs what #334 found missing, a signature list with its own sync test, and it adds to the API rather than changing it, so the core API freeze (2026-10-28 #925 (Derek), THE CORE API) lets any 3.x release take it. diff --git a/docs/design/decisions/H8-salutation-ending-the-given-part.md b/docs/design/decisions/H8-salutation-ending-the-given-part.md index 1702a273f..99e9bf436 100644 --- a/docs/design/decisions/H8-salutation-ending-the-given-part.md +++ b/docs/design/decisions/H8-salutation-ending-the-given-part.md @@ -8,7 +8,7 @@ - A salutation that is someone's middle name reads as the title after a family comma, an accepted cost (`Doe, Jane Miss`); so does the empty head's one given word moving to the family as the period-marked spelling moves it (`, Jane Mrs` → family 'Jane') — 2026-10-24 #835. - The rule reads the comma, not a declared order: `Doe Jane Mrs` under FAMILY_FIRST keeps middle 'Mrs', so the family-comma invariant of decisions/two-input-invariants does not hold for a trailing salutation — 2026-10-24 #835. -Open: [#877](https://github.com/derek73/python-nameparser/issues/877) (whether a bare `I` or `V` the salutation leaves ending the given part, read as the suffix by rules/C9, should report `suffix-or-name`). +Open: none in this file. Whether a bare `I` or `V` the salutation leaves ending the given part, read as the suffix by rules/C9, reports `suffix-or-name` was #877, decided in decisions/C9's 2026-11-04 #877 entry: it reports (`Doe, Jane V Mrs` → suffix 'V', reported). ## Log diff --git a/docs/design/decisions/H9-title-phrase.md b/docs/design/decisions/H9-title-phrase.md index 63ef948d0..aa0995c7a 100644 --- a/docs/design/decisions/H9-title-phrase.md +++ b/docs/design/decisions/H9-title-phrase.md @@ -7,7 +7,7 @@ - `grace` (a style) and `marshal` (a rank) are titles that may be given names, `sergeant` attaches to the word `major` and `major` to `general` (`attaches` naming a word where a kind would reach too far): `His Grace John Smith`, `Field Marshal Bernard Montgomery` and `Sergeant Major John Smith` read the title; `Mr Justice Smith` keeps given 'Justice', no exception behind a salutation — 2026-10-26 #752, phase 2i (Derek). - Its partner is a modifier or a style before it, a title before it that attaches to it or a title after it that it attaches to (`attaches`), or `of` after it; a salutation is never one, nor any other title before it (#853's `Mr Dean Smith`), nor a title after it it does not attach to (`SMITH, DEAN MR.`) — 2026-10-25 #752. - A modifier in the leading run of titles before the first comma is a title only with a head title after it in that run, `of` after it, or in the object of `of` or `the` there; outside that run (ending the name, H5's, or after a comma) it reads as any title does, the given part after a family comma included — 2026-10-25 #752; save a modifier that may be a given name, which the run's test reads in every part (`Lau, Wing Kai` → given 'Wing', `John Smith Wing.` → family 'Wing.') — 2026-10-26 #752, phase 2j, `wing`. -- A title that may be a given name which a caller lists in `titles` is a title wherever it stands, which keeps the 1.4 recipe `c.titles.add("dean")`; a v1 caller who empties `titles` turns the phrase reading off with them, while one who replaces the list keeps it (`c.titles.clear(); c.titles.add('dr')` still reads `Dean of Students` as the title), v1 holding no knob for the set — 2026-10-25 #752. A title 2.6.0 listed that a later batch made a title only inside a phrase stays in v1's default `titles`, read by the phrase while it is there and as no title once removed, so `c.titles.remove('king')` works as in 2.6.0; listing it in v1 cannot make it a title everywhere, which takes `Lexicon.default().add(titles={'king'})` — 2026-10-26 #752, the v1 title list. +- A title that may be a given name is a title wherever it stands once a caller takes its `given` reading out, which since 3.0 is `Lexicon.default().remove("dean", kind="given")`, the 1.4 recipe `c.titles.add("dean")` in 3.0's spelling (decisions/vocabulary-model's title warning names it) — 2026-10-25 #752. - `maj` is a title that may be a given name, attaching to `gen` and `general`, and `major` attaches to `gen` as to `general`: `Maj Sjöwall` reads given 'Maj', reported `title-or-name`, `Maj Gen John Smith` and `Major Gen John Smith` the title; a rank written short and bare at the head reads as the given name (`MAJ Ian Stewart, MD`), no capitals signal reading it — 2026-11-03 #999, `maj`. - `gen` is a title that may be a given name, written short, as `maj` is; the listed ranks and offices that stand before General attach to it, and `gen` and `maj` attach to `sir`: `Gen Goto` reads given 'Gen', reported `title-or-name`, `Lt Gen John Smith`, `Colonel Gen John Smith` and `Gen Sir Nick Carter` the title, `Gen. and Mrs. John Smith` the given name — 2026-11-03 #999, `gen`. - `brig` is a title that may be a given name, written short, a title word only since this entry, attaching to `gen`, `general` and `sir`: `Brig Gen Mark Kimmitt` reads the title, `Brig Klyce` and a bare `Brig John Smith` the given name, as before — 2026-11-03 #999, `brig`. @@ -15,6 +15,8 @@ - `lady` is a title that may be a given name only under a Spanish choice, es.toml's `given` reading being off by default, and `locales.ES` makes that choice; eight more titles borne as given names, `sir`, `general`, `bishop`, `colonel`, `canon`, `author`, `empress` and `lawyer`, stay titles everywhere, their counts on their title readings — 2026-11-03 #913, the last nine. - The article before any other title that may be a given name is no partner, save before a modifier, whose object of `the` decision 10 reads (`The Wing Smith` → title 'The Wing'), at the head, after a comma and under every order: `The Marquis de Sade` reads given 'The Marquis', `The Marquis of Tavistock` the title — 2026-11-03 #913, the article (Derek). +Not standing: what a v1 caller's `titles` list did to a title that may be a given name — emptying it, replacing it, removing a word 2.6.0 listed — overtaken by 3.0's removal of the v1 layer, #357 in decisions/3-0-reevaluations — 2026-10-25 #752, phase 2i; 2026-10-26 #752, the v1 title list. + ## Log - **2026-10-25 #752, phase 2i (its report in a family comma's given part, and its costs `His Grace John Smith` and `Field Marshal Bernard Montgomery`, superseded by 2026-10-26 #752, phase 2i (Derek); its cost `Lau, Wing`, by 2026-10-26 #752, phase 2j, `wing`) — A TITLE THAT MAY BE A GIVEN NAME IS A TITLE ONLY INSIDE A TITLE PHRASE, AND A MODIFIER AT THE HEAD OF THE NAME ONLY WITH A HEAD BESIDE IT.** Carries out items 10, 11 and 12 of decisions/vocabulary-model's 2026-10-23 #752 entry (decision 10 at the head of the name only; the phrase partner; #853's word after a title), measured here on the rule as built rather than on the spike those items were decided on. diff --git a/docs/design/decisions/O6-two-family-names.md b/docs/design/decisions/O6-two-family-names.md index 3755c5eab..43c868ea4 100644 --- a/docs/design/decisions/O6-two-family-names.md +++ b/docs/design/decisions/O6-two-family-names.md @@ -18,8 +18,9 @@ - The veto stays a clause of this rule though it reads a second signal, the taken word looked up in a list, and the rule is a split candidate — 2026-10-30 #959, decided. - Given-first, a particle run and the one word behind it whose `given` reading is written `after_particle` (rules/P10), standing beside a given name of one word, is not taken; under a family-first order it is, completing no given name there. Neither side of a join holding a name reading written `shared` (Sikh `kaur`, `singh`) is joined: each is no family name, and with the switch alone the name reads as with it off — 2026-11-03 #704, decided in chat. - The Spanish list, which `locales.ES` carries as its lexicon fragment since sub-phase 3g (a caller may pass a list of its own), is generated, not per-word vocabulary: INE's 2025 census tables, pooled over every name borne in Spain, cut at a measured rank (1,000 words, 90% given use) by `tools/es_given_names.py`; a change to it is a cutoff change — 2026-10-30 #959. +- A list per region, or another selection rule, for any Spanish-speaking country but Argentina is declined, each staying on `locales.ES`: in every group of at least 100 records measured, ES_AR's longer list beats ES's by at most 0.61 point on ORCID and no combination of the shipped switches beats `locales.ES` by more than 0.06 point, far below the country-pack measure's 2 points, and a list pooled over six countries came within half a point of per-country lists everywhere but Argentina and Peru — 2026-11-05 #1079 (Derek). -Open: why the Spanish list does less in Latin America, and whether a list per region or another selection rule would serve it better than the one list `locales.ES` carries, Argentina's answered by `locales.ES_AR` and the rest led by the 2026-11-04 entry's table, where ES_AR's list beats ES's in most of Latin America outside Mexico, Peru and the Caribbean (the 2026-10-30 #959 entry's Latin America bullet; the 2026-10-30 #753 phase 3g entry measures the pack against the default there) — #959. A name written with ñ dropped (`Begona`, `Inigo`) matches no entry, the key keeping the tilde; whether to fold ñ for data that strips it — #959. +Open: whether to fold ñ for data that strips it, a name written with ñ dropped (`Begona`, `Inigo`) matching no entry of the given-name list, whose key keeps the tilde — #1081. ## Log @@ -187,3 +188,9 @@ Open: why the Spanish list does less in Latin America, and whether a list per re - **ES_AR's list in the rest of Latin America.** Every Spanish-speaking group of at least 100 records under `locales.ES` and `locales.ES_AR`: ES_AR's list reads more in Chile, Colombia, Honduras, Nicaragua, Panama, Paraguay, El Salvador, Uruguay and Venezuela on ORCID (by 0.04 to 0.61 points), and in most PatentsView and PSC groups there, and less in Spain, Mexico, Peru, Cuba, the Dominican Republic, Puerto Rico, Bolivia, Costa Rica, Ecuador and Guatemala (by 0.11 to 1.48 points on ORCID). None of those passes the 2-point measure, so they stay on ES; the table is the Open line's lead. - **The shape.** A country pack is a `Locale` like any other: `languages` a region tag (`es-AR`), which selects its language's files by the BCP 47 ancestry a choice already follows, no `es-AR.toml` existing until a reading is Argentina's; the language's `[rules]` named by `rules_patch("es")`; and its own lexicon fragment. Declined: `[rules]` inheriting along a tag's ancestors, which no country measured needs (none turns a rule differently from its language), and which would need a table to write `two_family_names = false`, which the build's check 12 refuses because a pack's value overrides the caller's, and would make `parser_for(ES, ES_AR)` a scalar conflict decided by argument order; a separate preset or profile API, a second way to choose what a `Locale` chooses. The 2026-10-30 phase 3g entry's Argentine figures (19,377 ORCID records, 82.5% and 89.3%) and these (19,364, 83.44% and 90.19%) are two measurements: `score` leaves out the 13 records whose given box opens with a title word, as `gaps` does, and the tree is four days and several changes later (#959's hyphen count, #1057 and #1058). Measured 2026-11-04 on master at 0705d6a4 (#1058 merged) and on the branch for the shipped pack, Python 3.11, ORCID's extract of 2026-10-27 at 20,000 records a country, PSC's fetch recorded on #849, PatentsView's cache. Recompute: `uv run --with openpyxl --with xlrd python tools/es_given_names.py --top N --share S` prints a list (its second column the words); `uv run python tools/differential/config_diff.py score default plain:ES plain:ES_AR --languages es` scores the packs, `--switches two_family_names,two_given_names` the switches (and `plain:PT plain:PT_BR --languages pt,pt-BR`, `plain:fil --group PH --group Filipino`, `default --cap 5000 --min 1000` for every group), and a list as an `expr:parser_for(Locale(code="es", lexicon=Lexicon(given_names=open("LIST").read().split()), policy=locales.ES.policy, languages=frozenset({"es"})))`, `LIST` the tool's second column saved one word a line (`| awk '{print $2}' > LIST`). The comparator for a list is `plain:ES`, which the tree ships, never the list being measured. +- 2026-11-05 #1079 (Derek) — A LIST PER REGION, OR ANOTHER SELECTION RULE, FOR ANY SPANISH-SPEAKING COUNTRY BUT ARGENTINA IS DECLINED; EACH STAYS ON `locales.ES`. It answers the question three earlier passages point at the Open line for: the 2026-10-30 #959 entry's Latin America bullet (the #961 review, "is Open above"), the 2026-10-30 #753, phase 3g entry ("the Open line on Latin America stays a question about a better list") and the 2026-11-04 #1061 entry's rest-of-Latin-America bullet ("the Open line's lead"). The Open line itself no longer holds that question: #1079 cut it to the ñ question, #1081. Decided in chat during the 3.0 release sweep, on the figures already recorded. + - **ES_AR's list passes nowhere else.** By the 2026-11-04 #1061 entry's rest-of-Latin-America bullet, ES_AR's list reads more than ES's in Chile, Colombia, Honduras, Nicaragua, Panama, Paraguay, El Salvador, Uruguay and Venezuela on ORCID by 0.04 to 0.61 points, and less in Spain, Mexico, Peru, Cuba, the Dominican Republic, Puerto Rico, Bolivia, Costa Rica, Ecuador and Guatemala by 0.11 to 1.48 points; none passes that entry's 2-point measure, and ES_AR's list reads more in most PatentsView and PSC groups there too. Argentina, which passes it, has `locales.ES_AR`. The measure covered every Spanish-speaking group of at least 100 records; the decline reaches the smaller ones too, on no figure of their own. + - **Per-country lists did little better.** #959 found a list pooled over six countries within half a point of per-country lists everywhere but Argentina and Peru (the 2026-10-30 #959 entry's pooled-list bullet). + - **In Peru, little is left to gain.** On ORCID PE, 19,471 records, the switch alone reads 96.0%, "leaving little to fix" (the #961 review), and the switch with the Spanish list 95.9%, the list fixing 178 and breaking 189 (the 2026-10-30 #959 entry's table), a net loss its breaks spread over the whole list. #959's per-country list for Peru, learned from Peru's own boxes, broke 58 of the 9,735 records of its held-out half against 102 for the six-country pooled list (the same entry's pooled-list bullet); what it fixed is not recorded, and no per-country list was scored by the country-pack measure. + - **No switch combination passes.** By the 2026-11-04 #1061 entry's country-pack measure, no combination of the shipped switches beats `locales.ES` in any Spanish-speaking group but Argentina by more than 0.06 point on ORCID, the most being Peru's, far below its 2-point bar. + - **Where the list does help.** PSC's Peruvian filers gain with it, 81.0% → 83.6% of 116 records (the 2026-10-30 #753, phase 3g entry's table), which `locales.ES` already gives them. diff --git a/docs/design/decisions/S6-case-signal-at-the-suffix-slot.md b/docs/design/decisions/S6-case-signal-at-the-suffix-slot.md index d68c4fe59..110266185 100644 --- a/docs/design/decisions/S6-case-signal-at-the-suffix-slot.md +++ b/docs/design/decisions/S6-case-signal-at-the-suffix-slot.md @@ -14,7 +14,7 @@ Elsewhere: the case a generation word that is also a name reads (rules/S6's bullet on it) is decisions/S11's 2026-10-25 #752 entry. -Open: #965 (a listed credential written in its own mixed-case spelling, `FREng`, reads as the name, as the surname `Freng` does; decisions/suffix-acronym-collisions' entry of 2026-10-30 has the cost). +Open: #1053, carried over from #965 (a listed credential written in its own mixed-case spelling, `FREng`, reads as the name, as the surname `Freng` does; decisions/suffix-acronym-collisions' entry of 2026-10-30 has the cost). ## Log diff --git a/docs/design/decisions/S8-credentials-trailing-run.md b/docs/design/decisions/S8-credentials-trailing-run.md index 7933aa070..ae8c7f25d 100644 --- a/docs/design/decisions/S8-credentials-trailing-run.md +++ b/docs/design/decisions/S8-credentials-trailing-run.md @@ -18,7 +18,7 @@ Not standing: the split of 2026-10-17 #682, carried out. Elsewhere: the split `Ph. D.` starting the run in a main segment too, from where it stands though the walk leaves it out, and any other credential split across a space starting it as the one member of the ambiguous class that does (`John Smith Ph. D. Jones` → suffix 'Ph. D. Jones', reported; `John Smith Pharm D Jones` → suffix 'Pharm D Jones') — decisions/S12's 2026-10-25 #752 entry; a credential opening the part after the comma, and the split `Ph. D.` starting the run as `PhD` does (`Smith, John Ph. D. Jones` → suffix 'Ph. D. Jones') — decisions/C13's 2026-10-04 (Derek), #603 entry; the run stopping short of a lone letter ending the given part ahead of a further comma (`Smith, John PhD I, Jr.` → middle 'I', suffix 'PhD, Jr.') — decisions/C9's 2026-10-25 #819 entry. -Open: #965 (an unlisted word in capitals joins a listed credential's run behind it, `CBE XYZ`, but not in front of it, `XYZ CBE`). +Open: #1053, carried over from #965 (an unlisted word in capitals joins a listed credential's run behind it, `CBE XYZ`, but not in front of it, `XYZ CBE`). ## Log diff --git a/docs/design/decisions/differential-ledger.md b/docs/design/decisions/differential-ledger.md index ba0b11702..df7cbd225 100644 --- a/docs/design/decisions/differential-ledger.md +++ b/docs/design/decisions/differential-ledger.md @@ -9,7 +9,7 @@ Archived 2026-10-19 under rule 3 of the decisions folder's entry conventions (#7 - `tools/differential/snapshot.jsonl` records every corpus name's reading, one line per (name, order): the seven role fields, the ambiguity kinds, `initials()` and the two case-repair views since #826 (decisions/R4); `tests/v2/test_snapshot.py` fails on drift and prints the moves grouped by changed field, and a change that moves readings re-records, so the PR diff of the snapshot is the list of names it moved. Nothing is classified, and a reviewer can approve a large diff without reading it, the grouping being the mitigation; the ledgers, their guards and the contract/radar tiers are gone, trimming them having been declined — 2026-10-13 #664, a committed snapshot; the `facade` key left with `HumanName` — 2026-10-28 #357. - `compare.py` stays, unclassified: it runs a released wheel against the tree behind the version tell, the tree-side path check and a stripped `PYTHONPATH`, prints the snapshot's grouped report and exits 0. It answers how a release tagged before the snapshot read the corpus names, and the 1.4 promise is a question asked of it (`--baseline 1.4.0`), not a gate; for a later release `snapshot.py --since ` is the cheaper answer — 2026-10-13 #664, `compare.py` stays. - The CLDR agreement ratchet stays beside the snapshot, answering agreement with a corpus's stated fields rather than stability (`tests/v2/test_cldr_agreement.py`) — 2026-10-13 #664, the CLDR agreement ratchet. -- `Policy`'s fields are annotated with what they store, not widened to every input the constructor accepts, since every reader would see the union; a `.pyi` stub typing `__init__` wide and the attributes narrow is a 3.0 candidate (decisions/3-0-reevaluations) — 2026-07-24 #332. +- `Policy`'s fields are annotated with what they store, not widened to every input the constructor accepts, since every reader would see the union; a `.pyi` stub typing `__init__` wide and the attributes narrow is unbuilt, a 3.x candidate (decisions/3-0-reevaluations's #1079 entry) — 2026-07-24 #332. - The tree side of a comparison must import this checkout's source package, `is_relative_to( / "nameparser")` and not anywhere under the repository, whose `.venv/`, `build/` and `dist/` can hold a released wheel; `compare.py` asks `snapshot.check_source` for it — 2026-07-24 #332. - `DEFAULT_BASELINE` rotates at every release as a step of the AGENTS.md release checklist, not as an issue — 2026-07-24 #332. diff --git a/docs/design/decisions/ma-do.md b/docs/design/decisions/ma-do.md index 464f6ec3e..689bd2e6d 100644 --- a/docs/design/decisions/ma-do.md +++ b/docs/design/decisions/ma-do.md @@ -12,7 +12,7 @@ Elsewhere: the rest of the case reading — Title-case `Ma` and `Do` leaning sur Elsewhere: after the name, a member whose writing settles it is the suffix, not the nickname (`John Smith (MA)`, `John Smith (M.A)`; `JOHN SMITH (MA)` and `John Smith (Ma)` keep the nickname) — decisions/S13's 2026-10-26 #810 entry, which supersedes the line above in part. -Open: #792 (S1 and S8: whether a bracketed generation ends the credential run, `W. Russell (Jr.) Withers`), the neighbour of the parenthesized line above, which it does not move: `John Smith (MA) Withers` reads nickname 'MA', family 'Withers'. +Open: none in this file. Whether a bracketed generation ends the credential run was #792, decided in decisions/S1's 2026-11-04 #792 entry (`W. Russell (Jr.) Withers` → family 'Withers', suffix 'Jr.'), which leaves a bracketed member of this set as the parenthesized line above has it: `John Smith (MA) Withers` reads nickname 'MA', family 'Withers'. ## Log diff --git a/docs/design/decisions/suffix-acronym-collisions.md b/docs/design/decisions/suffix-acronym-collisions.md index 1d3128cd2..89ba0343e 100644 --- a/docs/design/decisions/suffix-acronym-collisions.md +++ b/docs/design/decisions/suffix-acronym-collisions.md @@ -33,7 +33,7 @@ Not standing: the #454 bullet's note that the `fix(suffix-routing)` ledger rule Elsewhere: since phase 2a the ambiguous marking above is derived from a word's name reading rather than written as a flag, and no membership above moved — decisions/vocabulary-model's 2026-10-25 #752 — A CLASH WITH A NAME IS A NAME READING entry (phase 2a). -Open: #965 (telling the credential `FREng`, written in mixed case, from the surname `Freng`; a case reading of S6's, held in decisions/S6). +Open: #1053, carried over from #965 (telling the credential `FREng`, written in mixed case, from the surname `Freng`; a case reading of S6's, held in decisions/S6). ## Log diff --git a/docs/design/decisions/vocabulary-model.md b/docs/design/decisions/vocabulary-model.md index 62b6bd983..4a4b08cfa 100644 --- a/docs/design/decisions/vocabulary-model.md +++ b/docs/design/decisions/vocabulary-model.md @@ -7,9 +7,9 @@ The vocabulary was a set of base sets plus marker subsets (`particles` and `part - The vocabulary redesign comes before adding many packs: it is designed for 3.0, and what is safe is pulled forward in four phases, (1) the data format and a compiler, snapshot-neutral, (2) derived ambiguity with the gate on every reading kind, (3) pack semantics, (4) the rules menu, order and shapes, and the output model — 2026-10-15/16. - Each word is described by its readings (word → reading → properties), each kind belonging to one category that decides the compiled field it feeds; detailed kinds are recorded even where the kinds of a category behave alike, generation and credential being separate kinds among the suffix category's six (`generation`, `credential`, `honorific`, `professional_honorific`, `status`, `courtesy`) — 2026-10-15/16; 2026-10-18 #737. - A language file holds the words the parser might meet in names written in that language, immigrant particles included, not the words of its etymology, and a word sits in the file of every language whose names it occurs in, a Latin transliteration in its source language's file; `mul.toml` holds the words shared on purpose instead (the generations `i` to `v`, `2` and `jr` with `junior` and `jnr`, `&`, the European titles below, and the degrees and salutations every country's data writes alike), a word joining it where it harms no language and would otherwise be duplicated into every language's file — 2026-10-15/16; 2026-10-18 #737; 2026-10-19 #749; 2026-10-19 #750, title kinds; 2026-11-02 #1005 (Derek); 2026-11-03 #1005, the strict-gap batches (Derek). -- The default loads every language and keeps working for names of any language. Choosing a language is to narrow it in phase 3 (#753): a pack is to settle only the clashes where its language has a vocabulary reading, a strict choice to read only the chosen languages' files, `mul.toml` and native-script vocabulary, and a pack's preferred reading to be able to outrank one a shape rule derives — 2026-10-15/16; 2026-10-30 #753, file-strict; 2026-11-02 #1005 (Derek). +- The default loads every language and keeps working for names of any language. Choosing a language narrows it since 3.0's phase 3 (#753): a pack settles only the clashes where its language has a vocabulary reading, a strict choice reads only the chosen languages' files, `mul.toml` and native-script vocabulary, and a pack's preferred reading can outrank one a shape rule derives — 2026-10-15/16; 2026-10-30 #753, file-strict; 2026-11-02 #1005 (Derek). - `e`, `y` and `i` stay in the default; choosing a language settles their one-case fork for that language's own reading, `may_be_initial` not holding on a chosen file's conjunction, as `locales.PT` and `locales.PT_BR` do for `e` — 2026-10-15/16; 2026-10-29 #753, phase 3b; 2026-10-30 #753, phase 3g. -- A patronymic field is to come in 3.0, as a view that names its words without emptying `family`, and the particle kinds already draw the `clan`/`patronymic` line for it: a patronymic is followed by a parent's name, a clan prefix by a family name the lineage shares — 2026-10-15/16; 2026-10-19 #749; 2026-10-28 #754 (Derek); landed as rules/R9, 2026-10-29 #754, sub-phase 4c. +- A patronymic field came in 3.0, as a view that names its words without emptying `family`, and the particle kinds already draw the `clan`/`patronymic` line for it: a patronymic is followed by a parent's name, a clan prefix by a family name the lineage shares — 2026-10-15/16; 2026-10-19 #749; 2026-10-28 #754 (Derek); landed as rules/R9, 2026-10-29 #754, sub-phase 4c. - A shipped pack that is only a choice of languages declares no `DEVIATES`, its surface the words its choice changes; `ga` turns on `nic`, `mhic`, `mic` and `uí` as clan particles, `gd` `nic`, and `pt-BR` `sb` as Sobrinho's spelling, while the single letters `f` and `n` are excluded, the pack included, the letter abbreviating a surname more often than the agnomen in the filings that write it — 2026-10-30 #753, phase 3h. - The Breton article `ar` is a reading of `br.toml` written `default = false`, read by a choice of `br` that no pack makes, and `M.` (Monsieur) is excluded, a French choice included, the letter an initial in every name data source measured; fr `père` is read by default (decisions/generational-words), so no French pack ships — 2026-11-03 #658. - `es` turns on the courtesy, academic and professional titles that lead a Spanish name with no complement (`lic`, `licenciado`, `licenciada`, `lcdo`, `lcda`, `ingeniero`, `ingeniera`, `arq`, `mtro`, `mtra`, `doctora`, `doña`) and `pt` `doutor`, `doutora`, `professora` and `profa`, `pt-BR` with it; `don`, `dona` and `dom` are excluded, the packs included, names opening with each as a name more often than as the title; `maestro`, a family name whose form of address is written `Mtro.`, is excluded too, and so is `h`, the hijo an initial is written as; `sr` keeps a generation reading beside its salutation in both files, so a choice leaves it the dual position decides; no connective continues a title run, the shape unwritten in name data, and the office titles that take a complement stay out — 2026-10-30 #753, phase 3g. @@ -21,9 +21,9 @@ The vocabulary was a set of base sets plus marker subsets (`particles` and `part - A reading right in its own language and wrong only as a default, because another language written in the same script collides with it, is written `default = false` in its file, never excluded, so that after the packs an exclusion means wrong everywhere: cs `st` since phase 2h, and cs `ml`, hr, sk and sl `ml` and `st` since 3c (fr `père`, converted with them, is read by default since 2026-11-03, a collision inside French), the rest in their packs' PRs — 2026-10-27 #753 (Derek); 2026-10-29 #753, phase 3c. - A name reading written `default = false` derives no marker in the default and counts for no check guarding the default's compile; a particle whose word's only `given` readings are off is refused until a pack needs one — 2026-10-29 #753, phase 3a. - A caller configures vocabulary by reading, `Lexicon` deriving the sets and gates the build derives, and the compiled sets become internal in 3.0 — 2026-10-28 #754, the `Lexicon` surface (Derek). -- The configure-by-reading API: `add(*words, kind, **properties)` and `remove(*words, kind=None)`, the views reading a word's kind from the lexicon, `readings_of` returning a public `LexiconReading` and a repr counting readings by kind, a five-part state (base, ordered edit log, choice, lists, masks) compared structurally, a clash a later choice exposes raised at `parser_for` naming the choice, the choice recorded on the `Lexicon` with chained choices unioning, the default's base lazy, a 2.x pickle translated onto the nearer named base and a per-word log, old spellings refused with a per-word translation, built in 4i-a to 4i-d with no default parse moving, 4i-a to 4i-c built — 2026-11-02 #754, the `Lexicon` surface (Derek); 2026-11-02 #754, sub-phase 4i-a; 2026-11-02 #754, sub-phase 4i-b; 2026-11-02 #754, sub-phase 4i-c. +- The configure-by-reading API: `add(*words, kind, **properties)` and `remove(*words, kind=None)`, the views reading a word's kind from the lexicon, `readings_of` returning a public `LexiconReading` and a repr counting readings by kind, a five-part state (base, ordered edit log, choice, lists, masks) compared structurally, a clash a later choice exposes raised at `parser_for` naming the choice, the choice recorded on the `Lexicon` with chained choices unioning, the default's base lazy, a 2.x pickle translated onto the nearer named base and a per-word log, old spellings refused with a per-word translation, built in 4i-a to 4i-d with no default parse moving — 2026-11-02 #754, the `Lexicon` surface (Derek); 2026-11-02 #754, sub-phase 4i-a; 2026-11-02 #754, sub-phase 4i-b; 2026-11-02 #754, sub-phase 4i-c; 2026-11-02 #754, sub-phase 4i-d. - A whole kind is taken out with `remove(kind=..., every=True)`, every word's readings of it going, those a later choice turns on included, and `words(kind=..., **properties)` lists the words holding a reading of a kind or category — 2026-11-02 #1007, a kind as a whole (Derek). -- Configured by reading, an `add` of a title reading that a `given` reading keeps a title only inside a phrase is to warn, naming `remove(w, kind="given")`, save one that only updates a held title reading's properties — 2026-11-01 #754, the title warning (Derek). +- Configured by reading, an `add` of a title reading that a `given` reading keeps a title only inside a phrase warns, naming `remove(w, kind="given")`, save one that only updates a held title reading's properties — 2026-11-01 #754, the title warning (Derek). - A 2.x call or pickle that removed a word from its base set but left it in a marker subset reads as though the marker went with the word, for every marker set, the two readings 2.x gave such an orphan accepted as moving — 2026-11-02 #754, a marker left behind. - A `Locale` may carry CLDR-shaped render patterns, display and sorting, a later addition whose shape is agreed; how a call reaches one is open — 2026-10-28 #754, render patterns. - 3.0's output model keeps the seven roles and adds every new field as a view, read from `Token.reading`, the reading each token took, or for `surname2` from rules/O6's tag: `generation`, `credentials` and, since 2026-11-03, `honorifics` partitioning `suffix` (decisions/R8), `patronymic`, and `surname2` where rules/O6 read two family names; `middle` and `surnames` keep their meaning, a compound given name being a rule a caller or pack turns on; the default `str()` renders a name in the order it was read in with its written separators in a normalized form, through `render(spec: str | None = None)`, `None` being that default; a language file's `[rules]` compile into its `Locale`'s policy patch — 2026-10-28 #754 (Derek). @@ -32,7 +32,7 @@ The vocabulary was a set of base sets plus marker subsets (`particles` and `part - Title-phrase modifiers count as titles only beside a head title, at the head of the name only and never in a comma part (`Wing Lau` → given 'Wing', `John Smith, Vice` → title 'Vice'), save a modifier that may be a given name, which the head's test reads in every part (`Lau, Wing Kai` → given 'Wing'), rules/H9 — 2026-10-15/16; 2026-10-19 #750, title kinds; 2026-10-23 #752; 2026-10-25 #752, phase 2i; 2026-10-26 #752, phase 2j, `wing`. - Name readings are hand-declared, and only on a word some language file claims as vocabulary or refuses, so the vocabulary carries no surname list (the segmentation lists, Korean and the ZH pack's, are outside the per-word model) — 2026-10-15/16; 2026-10-19. - Bearer counts are structured provenance (source and count), never used to compute a reading — 2026-10-15/16. -- Phase 2 is to put vocabulary, declared name and shape readings in one pool, derive `may_be_given` and `may_be_name` from declared name readings (a count-free one carrying a `note`, `ld` left ungated, `may_be_initial` kept), and decide every clash by one gate of four steps, floor, form, structure and count, asked at the sites that decide today; the reading taken is to surface as `Token.reading` in 3.0 — 2026-10-23 #752. The reading not taken surfaces as `Ambiguity.alternative` since 2.7 (decisions/A1) — 2026-10-26 #752, phase 2m. +- Phase 2 put vocabulary, declared name and shape readings in one pool decided by one gate of four steps, floor, form, structure and count, asked at the sites that decide, four clashes and two floors (the 2026-10-23 entry's item 4), and derived `may_be_given` and `may_be_name` from declared name readings (a count-free one carrying a `note`, `ld` left ungated, `may_be_initial` kept) — 2026-10-23 #752. It shipped piecemeal, 2a to 2m, a credential's capitals landing through the pool — 2026-10-25 #752, phase 2g; 2026-10-26 #752, phase 2m. A configured `Lexicon` compiles its pool from its base's rows and its log since 3.0 — 2026-11-02 #754, sub-phase 4i-b. The reading taken surfaces as `Token.reading` since 3.0 — 2026-10-29 #754, sub-phase 4a. The reading not taken surfaces as `Ambiguity.alternative` since 2.7 (decisions/A1) — 2026-10-26 #752, phase 2m. - Every reading the parse took or a report names from the vocabulary names the shipped files holding it, `Reading.languages`, looked up at assemble in the readings of the parse's `Lexicon` (the reading index's rows for the default) by the word and the kind, a name by the field it lands in; the files whose reading is on in the `Lexicon` the parse used, so a choice names the files it reads and not those it drops, and none for a shape, position, a caller's own reading, or a word no file holds in that category — 2026-10-29 #754, 4a's follow-up; 2026-11-02 #754, sub-phase 4i-d. - Every numeral rule asks one predicate, `_vocab.reads_as_generation`, the vocabulary's own numerals read inside it; a numeral of two or more letters is the generation wherever the trailing run holds it, while a one-letter one before a credential stays the name word decisions/S10's #721 entry made it — 2026-10-25 #752, phase 2f. - A credential written split across a space is one word, a member of the ambiguous class that the count decides, and the Ph./D. pair starts S8's run from where it stands (decisions/S12) — 2026-10-25 #752, phase 2e. @@ -48,7 +48,7 @@ The vocabulary was a set of base sets plus marker subsets (`particles` and `part - Exclusions are data: an `excluded` key refuses a kind, a category or a behavioral property, in the file of the language whose reading it refuses, and the build refuses the excluded thing in any file, quoting the reason — 2026-10-18 #737. - A per-word fact that moves into the TOML is replaced in decisions/ by a query that finds it (decisions/README.md, edit kind 4) — 2026-10-18 #737. - The script is read from the word, not the file: a script subtag is only for a difference the text cannot show — 2026-10-18 #737. -- The names later phases will use (`prefer`, `strict`, the position and attachment keys) are build errors naming their phase, not inert data; locale-pack vocabulary and the Korean surname list stay in Python — 2026-10-18 #737. `bare_only`, `leading_only`, `trailing_only` and `separators` are phase 4's; `default = false` records a reading that compiles to nothing, for a pack to turn on, and no exclusion refuses it (Spanish `h`, German `i` for the dotted regnal ordinal, Czech `st`) — 2026-10-25 #752, phase 2h. `attaches` names the title kinds, or the title words, a title reading stands before in a title phrase, compiled into the phrases it makes — 2026-10-25 #752, phase 2i; 2026-10-26 #752, phase 2i (Derek). +- The names reserved for a later use are build errors, not inert data; locale-pack vocabulary and the Korean surname list stay in Python — 2026-10-18 #737. `prefer` and `strict` are refused for good as data, a strength being the caller's — 2026-10-31 #753, phase 3j. `bare_only`, `leading_only`, `trailing_only` and `separators` stay reserved, unbuilt, for 3.x, their build message naming the release — 2026-11-05 #1079; `default = false` records a reading that compiles to nothing, for a pack to turn on, and no exclusion refuses it (Spanish `h`, German `i` for the dotted regnal ordinal, Czech `st`) — 2026-10-25 #752, phase 2h. `attaches` names the title kinds, or the title words, a title reading stands before in a title phrase, compiled into the phrases it makes — 2026-10-25 #752, phase 2i; 2026-10-26 #752, phase 2i (Derek). - A title reading beside a `given` reading, on a word with no particle reading, derives `TITLES_AMBIGUOUS` instead of `TITLES`: a title only inside a title phrase (rules/H9); a salutation's given-name counts kept as evidence for H8 sit in its `note`, marking nothing — 2026-10-25 #752, phase 2i. - Every language file is in build-checked order (kind, then the `same_as` path from its root head, then the codepoints of the text), the check naming the entry to move, running last, and refusing a `same_as` loop of any length; `und.toml` is exempt — 2026-10-19 #747; 2026-10-21 #762. - The particles take eight kinds beside the generic one, `expands` carrying a particle's gloss and `note` its vetting, and the particle keep-outs are exclusions — 2026-10-19 #749. @@ -580,4 +580,4 @@ Open: #753 (phase 3, language packs; the 16 words strict ungates, each read for - **2026-11-03 #980 — `es` TURNS ON `maestra`; `maestro` STAYS EXCLUDED; `profesor` AND `profesora` LEAVE THE CHOICE FOR THE DEFAULT.** The `dof-signatures` local corpus (#977), the signature blocks of Mexico's Diario Oficial for 2025, writes the master's-degree form of address spelled out 85 times (`Maestro` 35, `Maestra` 34, `MAESTRO` 12, `MAESTRA` 4, #980's count at the corpus's cut version 6; 86 in the fetch of 2026-11-03, `Maestra` 35), every one read without a title under `locales.ES` on master. `maestra` gets an `academic` reading written `default = false` in es.toml, beside `mtra`, which becomes its `same_as`: it is also the Spanish and Italian word for a schoolteacher and a family name with a few bearers (ORCID's family box 2 records in Spain, 1 in Italy; FEC 1, `MAESTRA, SUSAN M.`, a family comma that keeps it), no US Census, US SSA or Wikidata bearer (`tools/name_usage.py --boxes`, 2026-11-03). Under the pack `Maestra Ana Ruiz Paz` reads title 'Maestra', given 'Ana', family 'Ruiz Paz'. `maestro` stays excluded as phase 3g decided (a Spanish family name, ORCID's family box 43 records in Spain): #980 asked whether a title-phrase-only reading would serve both, and it would not, the DOF writing the bare form, and a title-phrase gate (rules/H9) is derived from a `given` reading, which neither word has. A complement behind it is not taken, the title-run connective phase 3g declined: `Maestra en Derecho Rosa María Millán Gómez` reads given 'en' under the pack, as 2 DOF names in 3 signatures are written (`Maestra en Derecho`, `MAESTRA EN ADMINISTRACION`). `profesor` and `profesora`, which phase 3g turned on with the choice, are read by default since this change (decisions/salutation-titles' 2026-11-03 #804, Spanish entry), so they leave the choice's deviation surface. - **What moves.** Under the default, only `profesor` and `profesora`, measured in the salutation-titles entry. Under `locales.ES` beyond that, `maestra` in the 38 DOF signatures it opens, of the 39 holding it (in `C. Maestra Olga Santillán Rodríguez` the parse reads `C.` as an initial, the given name, and `Maestra` stays a middle name). - **Recompute.** The signature counts by a scan of `dof-signatures`' `raw.jsonl`, one record a signature (its `corpus.jsonl` holds the 6,644 distinct names), for the word, case-folded, and its position; `tools/differential/config_diff.py diff plain:ES 'expr:parser_for(locales.ES, base=Parser(lexicon=Lexicon.default().remove("maestra", kind="academic")))' --local dof-signatures --match '(?i)\bmaestra\b'` in the tree reads the 14 distinct names it opens losing the title without it, the pack's two family names held alike on both sides. - +- **2026-11-05 #1079 — THE FOUR NAMES PHASE 4 WAS TO USE STAY RESERVED, UNBUILT, FOR 3.x, AND THEIR BUILD MESSAGE NAMES THE RELEASE RATHER THAN A PHASE.** 3.0 ships without phase 4's position and bare-letter properties and its glued words, so `bare_only`, `leading_only`, `trailing_only` and `separators` stay build errors, which now say each is "reserved, unbuilt, for 3.x" (`tools/vocab/build.py`'s `RESERVED`), where the 2026-10-25 #752, phase 2h entry had them name phase 4. nameparser/vocab/README.md's table of them follows, and tests/v2/test_vocab.py holds the two equal. Decided in chat during the 3.0 release sweep. diff --git a/docs/design/rules/C3-counting-the-words-before-the-comma.md b/docs/design/rules/C3-counting-the-words-before-the-comma.md index db76989b2..e438be96c 100644 --- a/docs/design/rules/C3-counting-the-words-before-the-comma.md +++ b/docs/design/rules/C3-counting-the-words-before-the-comma.md @@ -61,7 +61,7 @@ ``` "Dumas X, Jr." → family="X" ``` -- A title in front of a particle surname, before the credential class's word, reads the listing form. The surname counts once and the title is no name word, so the count reads the listing form, and the listing form peels a title before the comma (H6) unless the word is a particle too, as 'Freiherr' is, which stays in the family. For 'Freiherr' that reading is the right one: since 1919 a former noble title is part of the legal surname, written between the given name and the particle, so 'Freiherr von Berg' is the family name in 'Freiherr von Berg, Ed', as 2.0 through 2.3 read it. A rank that is no particle peels as a title like any other ('Graf Lambsdorff, Otto' reads title 'Graf'), though the same law makes it part of the surname too; #709 asks whether it should stay. Before an unambiguous credential the same words keep the title apart ('Freiherr von Berg, PhD'), that count taking the title as a word; both are readings of one name. +- A title in front of a particle surname, before the credential class's word, reads the listing form. The surname counts once and the title is no name word, so the count reads the listing form, and the listing form peels a title before the comma (H6) unless the word is a particle too, as 'Freiherr' is, which stays in the family. For 'Freiherr' that reading is the right one: since 1919 a former noble title is part of the legal surname, written between the given name and the particle, so 'Freiherr von Berg' is the family name in 'Freiherr von Berg, Ed', as 2.0 through 2.3 read it. A rank that is no particle peels as a title like any other ('Herzog Smith, Anna' reads title 'Herzog'), though the same law makes it part of the surname too; 'Graf' stays, a particle since #709. Before an unambiguous credential the same words keep the title apart ('Freiherr von Berg, PhD'), that count taking the title as a word; both are readings of one name. ``` "Freiherr von Berg, Ed" → family="Freiherr von Berg" ``` diff --git a/docs/design/rules/H9-title-phrase.md b/docs/design/rules/H9-title-phrase.md index 6c7bfe368..222cced7d 100644 --- a/docs/design/rules/H9-title-phrase.md +++ b/docs/design/rules/H9-title-phrase.md @@ -117,7 +117,7 @@ "Dean and Mary Smith" → given="Dean and Mary" "Dean. and Mary Smith" → given="Dean. and Mary" ``` -- **A title that may be a given name which a caller lists in `titles`** is a title wherever it stands, since that is what listing it asks; a modifier is read by this rule however it is listed, and leaving its word out of the modifiers turns the rule off for it. +- **A title that may be a given name whose given reading a caller removes** is a title wherever it stands, since that is what removing it asks; a modifier is read by this rule unless a caller reads its word as another title kind, and leaving its word out of the modifiers turns the rule off for it. **Accepted:** diff --git a/docs/design/rules/P9-particle-that-is-also-a-suffix-word-beside-a-family-comma.md b/docs/design/rules/P9-particle-that-is-also-a-suffix-word-beside-a-family-comma.md index 199c37c49..a16c0d4cf 100644 --- a/docs/design/rules/P9-particle-that-is-also-a-suffix-word-beside-a-family-comma.md +++ b/docs/design/rules/P9-particle-that-is-also-a-suffix-word-beside-a-family-comma.md @@ -9,6 +9,10 @@ "SMITH VD MA, JOHN" → family="SMITH VD MA" ``` - **A lone letter behind it reads as it does in the same part without the particle**: a capital is a name word ('SMITH VD V, JOHN' as 'SMITH V, JOHN'), a lowercase letter in a name of mixed case the numeral ('Smith vd v, John' as 'Smith v, John'), and a name written in one case reads as its capitals (T4). + ``` + "SMITH VD V, JOHN" → family="SMITH VD V" deviates: #1080 (today: family="SMITH VD") + "smith vd v, john" → family="smith vd v" deviates: #1080 (today: family="smith vd") + ``` - **In the part after a family comma**, where P6 attaches a particle ending the name to the family, a lone word of both the particle and the UNAMBIGUOUS suffix vocabulary standing INSIDE a run read as post-nominals — one in front of it and another behind, a credential or a generation, whatever made each one, and a title between it and the one in front transparent (H5) — is not the tussenvoegsel: that stands right behind the given name, so a post-nominal between them says the word is not one. It does not end the name, and keeps the post-nominal reading in any case, reported as S4's credential fork. ``` "DOE, JANE PHD VD MA" → suffix="PHD VD MA" diff --git a/docs/design/rules/S-suffixes-generational-and-credentials.md b/docs/design/rules/S-suffixes-generational-and-credentials.md index 9432243f8..bb7569826 100644 --- a/docs/design/rules/S-suffixes-generational-and-credentials.md +++ b/docs/design/rules/S-suffixes-generational-and-credentials.md @@ -1,4 +1,4 @@ ## Suffixes: generational & credentials (S) -Background: what follows a name is one of two different things — generational suffixes (Jr., III), which attach to the name itself, and credentials (PhD, MD, MBA), which are earned attachments. The suffix sets match one written word at a time: a multi-word entry there can never match anything and is warned about at configuration. Only three sets are exempt from that rule -- given_name_titles, maiden_markers since #434, and title_phrases since #894 -- and none is a suffix set, so within the suffix vocabulary the one-word rule is absolute. That limit is on STORAGE, not on the shape a name may have: adjacent suffix tokens are reassembled after matching (`_vocab.is_wholly_suffix`), so a multi-word credential is reachable as its component words -- `John Smith, MD PhD` has read suffix `MD PhD` since 1.4.0 -- and a caller reaches an unshipped one by adding the words it is made of rather than the phrase (#433). The eight multi-word entries that shipped dead for years span the suffix sets and the titles alike and are the Excluded story in decisions/vocabulary-exclusions. CLDR personNames keeps them as separate fields (`generation`, `credentials`) and formats them differently; this library reports both in one `suffix` field, and since 3.0 three views partition it by the reading each word took (R8, #326). The vocabulary is largely split already: a generational word list and a credential acronym list, plus a short list of acronyms that are also ordinary names (MA, BA) and so are AMBIGUOUS as bare words. +Background: what follows a name is one of two different things — generational suffixes (Jr., III), which attach to the name itself, and credentials (PhD, MD, MBA), which are earned attachments. The suffix sets match one written word at a time: a multi-word entry there can never match anything, and since 3.0 a `Lexicon` refuses one, raising at the `add()` that writes it. Only three sets are exempt from that rule -- given_name_titles, maiden_markers since #434, and title_phrases since #894 -- and none is a suffix set, so within the suffix vocabulary the one-word rule is absolute. That limit is on STORAGE, not on the shape a name may have: adjacent suffix tokens are reassembled after matching (`_vocab.is_wholly_suffix`), so a multi-word credential is reachable as its component words -- `John Smith, MD PhD` has read suffix `MD PhD` since 1.4.0 -- and a caller reaches an unshipped one by adding the words it is made of rather than the phrase (#433). The eight multi-word entries that shipped dead for years span the suffix sets and the titles alike and are the Excluded story in decisions/vocabulary-exclusions. CLDR personNames keeps them as separate fields (`generation`, `credentials`) and formats them differently; this library reports both in one `suffix` field, and since 3.0 three views partition it by the reading each word took (R8, #326). The vocabulary is largely split already: a generational word list and a credential acronym list, plus a short list of acronyms that are also ordinary names (MA, BA) and so are AMBIGUOUS as bare words. diff --git a/docs/design/vocabulary-model-rfc.md b/docs/design/vocabulary-model-rfc.md index a4363c727..c6b1e1627 100644 --- a/docs/design/vocabulary-model-rfc.md +++ b/docs/design/vocabulary-model-rfc.md @@ -139,7 +139,7 @@ A caller should be able to see the reading behind a parse. **An `Ambiguity` carr ### The phase 1 reserved names -Phase 2 gives two of them meaning: `attaches`, the phrase partner above, a property of a title reading naming the title kinds it qualifies; and `default = false`, for readings with no cue. `leading_only` and `trailing_only` are not needed, the gate's slots carrying position; `bare_only` is settled by shape already ([rules/P7](rules/P7-one-letter-particle-against-the-initial-it-spells.md), a one-letter particle against the initial it spells); `separators` belongs with `attached` and its other side, phase 4's glued words. Those four stay reserved, their messages naming the phase that needs them. +Phase 2 gives two of them meaning: `attaches`, the phrase partner above, a property of a title reading naming the title kinds it qualifies; and `default = false`, for readings with no cue. `leading_only` and `trailing_only` are not needed, the gate's slots carrying position; `bare_only` is settled by shape already ([rules/P7](rules/P7-one-letter-particle-against-the-initial-it-spells.md), a one-letter particle against the initial it spells); `separators` belongs with `attached` and its other side, phase 4's glued words. Those four stay reserved, their messages naming the phase that needs them (since 2026-11-05, #1079, the release instead: unbuilt, 3.x). ### The order of work @@ -583,7 +583,7 @@ Each sub-phase is one PR, measured on its own base as phase 2's and phase 3's ar **Interleaved with phase 3** (its sub-phases 3a to 3j, above): 4a can start beside 3a, its `languages` waiting for 3a's index; 4d needs nothing of phase 3, and 4b and 4c need 3a's index only for a listed word's kind, a suffix word's and a particle's; they unblock its packs, `cs`'s generations (3f) and `pt-BR`'s agnomens (3h) on 4b, `es` and `pt` (3g) on 4d and 4g; 4f lands with or after 3b, since both decide what a shipped `Locale` holds; an `is` pack, which no phase 3 sub-phase lists, needs 3b, 4c and 4f. 3j, file-strict, needs 3b alone (3i, which name-strict would have read, is set aside), and 4e and 4h stand apart from both. So the order is 3a and 4a, then 4b to 4d with 3b and 3c (4b after 3a), then 4f, then the packs with 4g, with 4e at any point and 3j at any point after 3b. -**Left open.** With no sub-phase yet: phase 1's four reserved names said to be phase 4's (`separators` with phase 4's glued words, and `bare_only`, `leading_only` and `trailing_only`, which this RFC's phase 2 section found not needed), whose build messages still name this phase. (The first unit's view of a family read as two, with its particles and base, was left open here until decisions/R11's 2026-11-03 #1022 entry built it.) +**Left open.** With no sub-phase yet: phase 1's four reserved names said to be phase 4's (`separators` with phase 4's glued words, and `bare_only`, `leading_only` and `trailing_only`, which this RFC's phase 2 section found not needed), whose build messages say each is reserved, unbuilt, for 3.x (decisions/vocabulary-model, 2026-11-05 #1079). (The first unit's view of a family read as two, with its particles and base, was left open here until decisions/R11's 2026-11-03 #1022 entry built it.) A shape question, decided 2026-10-28: diff --git a/docs/index.rst b/docs/index.rst index e85eb6814..28927b28f 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -23,7 +23,7 @@ packs are opt-in. Requires Python 3.11+. >>> name.render("{family}, {given}") 'de la Vega, Juan' -Coming from 1.x? :doc:`migrate`. +Coming from 1.x, or from 2.x code using ``HumanName``? :doc:`migrate`. .. toctree:: :maxdepth: 2 diff --git a/docs/locales.rst b/docs/locales.rst index 696084422..c3a728be9 100644 --- a/docs/locales.rst +++ b/docs/locales.rst @@ -1,14 +1,21 @@ Locale packs ============ -A locale pack is an opt-in bundle of policy — and, when a naming -tradition needs it, vocabulary — for one specific pattern: East Slavic -patronymics, Turkic patronymic markers, and so on. Packs apply only -when you ask for one by name, and every pack makes the same promise: -it never changes a name outside the shapes it declares. Read that -precisely — a shape is a pattern, not a language, so a pack can act on -a name from a tradition it was not written for. The warning under -`Using a pack`_ shows what that costs. +A locale pack is an opt-in bundle of policy, vocabulary and a choice +of languages, each where a naming tradition needs it, for one specific +pattern: East Slavic patronymics, Turkic patronymic markers, the +titles and particles of one language, and so on. Packs apply only +when you ask for one by name, and every pack is held to the same +test: over the test corpus and the pack's own example names, every +name it reads differently from the default is one its declared surface +covers. A rule's surface is a shape — a patronymic ending or marker +for ``ru`` and ``tr_az``, a script for ``zh`` and ``ja``, three name +words or more for the packs that read two family names — and a choice +of languages declares the words whose reading it changes. Read that precisely — a shape is a pattern, not a +language, so a pack can act on a name from a tradition it was not +written for, and a broad shape covers much: ``es``, ``es_ar``, ``cs`` +and ``sk`` may read any name of three words differently. The warning +under `Using a pack`_ shows what that costs. What works without a pack -------------------------- @@ -83,7 +90,9 @@ default: the ``zh`` and ``ja`` packs below turn it on for data whose language you can declare. A pack is for something different: a *structural* rule, like reordering -a patronymic, that vocabulary alone can't express. +a patronymic, that vocabulary alone can't express, or words that are +right in one language and wrong as a default (`Packs that choose a +language`_). Using a pack ------------ @@ -565,7 +574,9 @@ The command line accepts the same codes: ``python -m nameparser parsing, equivalent to ``parser_for(locales.get("ru"))``. ``--locale ja`` also attaches :func:`~nameparser.locales.ja_segmenter` where ``nameparser[ja]`` is installed, which ``parser_for`` never does on its -own (the ``ja`` row below). +own (the ``ja`` row below). A code no pack has but a vocabulary file +does (``--locale ko``) chooses that language alone, and ``--strict`` +makes the choice strict (`Reading one language only`_). Shipped packs ~~~~~~~~~~~~~ @@ -685,9 +696,9 @@ and nothing else: the readings each turns on are the default vocabulary's own, written in its language's file (``ne.toml`` for Nepali) and off until a caller chooses the language. ``ja``, ``ru`` and ``tr_az`` are policy-only — they carry no vocabulary of their own. -``es`` is a choice with a list and one policy field beside it: it turns -``two_family_names`` on and carries the given names that switch leaves -where they stand. ``cs`` and ``sk`` are a choice with that one policy +``es`` and ``es_ar`` are each a choice with a list and one policy field +beside it: each turns ``two_family_names`` on and carries the given +names that switch leaves where they stand. ``cs`` and ``sk`` are a choice with that one policy field beside it and no list. ``zh`` is both halves at once: a surname list, plus the one policy field that turns segmentation on for the script it covers. See :ref:`config-containers` for how that split (language vocabulary vs. behavior) @@ -795,6 +806,8 @@ readings of those words: - **Under pt_br,** a written S.B. degree reads as Sobrinho (``Howard Gordon, MD SB`` fills ``generation`` with ``SB``). +.. _spanish-portuguese-titles: + Spanish and Portuguese titles ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -1059,7 +1072,7 @@ by ``tests/v2/test_locales.py``: table in the pack's language file, list the file in ``PACK_RULES`` in ``tools/vocab/build.py``, rebuild, and give the pack ``policy=rules_patch("")`` from ``nameparser._locale``, as - ``ru`` and ``tr_az`` do. ``nameparser/vocab/README.md``'s "Rules" + ``ru``, ``tr_az``, ``es``, ``cs`` and ``sk`` do. ``nameparser/vocab/README.md``'s "Rules" section lists the keys. A rule the table does not list is a change to the parser, not to a pack. #. Declare a module-level ``DEVIATES(name)`` predicate. Given a name diff --git a/docs/migrate.rst b/docs/migrate.rst index f2608239a..7638b337d 100644 --- a/docs/migrate.rst +++ b/docs/migrate.rst @@ -20,8 +20,7 @@ are gone, a lexicon listing its words instead (`Vocabulary constants → Lexicon.words`_). Separately, the output model changes in 3.0: ``str()`` now renders a name in the order it was read in, with the separators it was written -with (`Changed in 3.0`_). Further output changes come with the 3.0 -alphas. +with (`Changed in 3.0`_). The full 1.x documentation remains the reference for ``HumanName`` and stays online as the 1.4.0 build on readthedocs: @@ -30,9 +29,11 @@ https://nameparser.readthedocs.io/en/v1.4.0/ What 3.0 removes ---------------- -2.8 warned at the line in your code that used each of these, and 2.8.1 -where code reads a constant of ``nameparser.config.regexes``, with the -exceptions `Finding the sites`_ names: +2.8 warned at the line in your code that used each of these, 2.8.1 +where code reads a constant of ``nameparser.config.regexes`` or a +vocabulary module, and 2.8.2 where it +reads ``KOREAN_SURNAMES``, with the exceptions `Finding the sites`_ +names: .. list-table:: :header-rows: 1 @@ -87,7 +88,7 @@ a ``Constants`` cannot load, because there is no class to load it into. Finding the sites ~~~~~~~~~~~~~~~~~ -Upgrade to 2.8.1 first. A ``DeprecationWarning`` is hidden by default +Upgrade to 2.8.2 first. A ``DeprecationWarning`` is hidden by default outside ``__main__``, so run your tests with warnings turned into errors, and each use 3.0 removes stops the run at the line that makes it: @@ -97,11 +98,12 @@ it: python -W error::DeprecationWarning -m pytest 2.8 warns where code *uses* a removed name, not where it imports one, -and 2.8.1 warns where code reads a constant of a removed module, +2.8.1 warns where code reads a constant of a removed module, ``nameparser.config.regexes`` or a vocabulary module (``from nameparser.config.titles import TITLES`` included), not where it -imports the module, so some code passes that check and still fails on -3.0: +imports the module, and 2.8.2 the same for +``nameparser.config.surnames``, so some code passes that check and +still fails on 3.0: * *reading* the shared ``CONSTANTS`` (``"dr" in CONSTANTS.titles``, or ``copy.copy(CONSTANTS)``), which 2.8 left quiet so that importing @@ -113,14 +115,18 @@ imports the module, so some code passes that check and still fails on importing ``nameparser.parser`` or the two 1.x word-list modules, ``nameparser.config.prefixes`` and ``nameparser.config.bound_first_names`` -* importing ``nameparser.config.regexes`` or a vocabulary module - without reading a constant from it — ``import +* importing ``nameparser.config.regexes``, ``nameparser.config.surnames`` + or a vocabulary module without reading a constant from it — ``import nameparser.config.regexes``, ``import nameparser.config.titles`` or ``from nameparser.config import particles`` +* *reading* a ``Lexicon`` set field (``"dr" in lexicon.titles``), which + 2.8.1 warns of only where code writes one, and which raises + ``AttributeError`` in 3.0 (`Lexicon fields → readings`_) So search your code for ``CONSTANTS``, ``HumanName``, ``Constants``, ``nameparser.parser``, ``nameparser.util``, ``config.prefixes``, -``config.bound_first_names`` and ``config.regexes`` as well, and for the +``config.bound_first_names``, ``config.regexes`` and ``config.surnames`` +as well, and for the vocabulary modules: ``config.titles``, ``config.suffixes``, ``config.particles``, ``config.conjunctions``, ``config.bound_given_names``, ``config.given_names``, @@ -1159,3 +1165,79 @@ first character: >>> parse("김민준").initials(), parse("Kim Minjun").initials() ('', 'K. M.') + +3.0 moved these 1.4 readings too. The release log's 3.0 section +carries the reasoning; these are the shapes to grep your fixtures for, +with what 1.4 read: + +- ``"Otto Graf Lambsdorff"`` — ``Graf``, ``Gräfin``, ``Fürst`` and + ``Prinz`` inside a surname join it, as ``Freiherr`` does: 1.4 read + middle ``Graf``, last ``Lambsdorff``. Opening a name each is still a + title (``"Graf Zeppelin"``). +- ``"Juan Sta. Maria Reyes"`` and ``"Juan Delos Reyes"`` — ``Sta.``, + ``Sto.`` and the Filipino ``Delos`` are surname particles: 1.4 read + middle ``Sta. Maria`` and ``Delos``, last ``Reyes``. +- ``"María del Carmen García"`` — a compound given name written with a + particle is a middle name: 1.4 read last ``del Carmen García``. +- ``"طارق عبد العزيز"`` — ``عبد`` joins a listed name of God after it: + 1.4 read middle ``عبد``, last ``العزيز``. +- ``"Ortega y Gasset VI"`` — 1.4 read first ``Ortega y Gasset``, last + ``VI``. +- ``"John Smith (PhD, MBA)"`` and ``"Justin (JP) Stearns"`` — a + bracketed clause of credentials is the suffix, and initials in + brackets the nickname: 1.4 read nickname ``PhD, MBA`` and middle + ``JP``. +- ``"Alexandre Dumas père"`` — 1.4 read middle ``Dumas``, last + ``père``. +- ``"Jean Dupont, Pr"`` and ``"Juan Pérez, Profesor"`` — the French and + Spanish professor titles are titles: 1.4 read first ``Pr`` and + ``Profesor``, last ``Jean Dupont`` and ``Juan Pérez``. +- ``"Jane Doe KC"`` and ``"Jane Doe MSW"`` — British post-nominals and + 77 common US credentials are suffixes: 1.4 read middle ``Doe``, last + ``KC`` and ``MSW``. +- ``"John⁣Smith"`` — the invisible separator divides words: 1.4 + read first ``John⁣Smith``. +- ``"Maj Sjöwall"``, ``"Gen Goto"``, ``"Pharaoh Mwale"`` and ``"Saint + Clair Soares"`` — a bare rank or ``Pharaoh``, ``Saint``, ``Sailor`` or + ``Majesty`` opening a name is the given name: 1.4 read title ``Maj``, + ``Gen``, ``Pharaoh`` and ``Saint``. ``"Brig Gen Mark Kimmitt"`` is one + title: 1.4 read first ``Brig``. +- ``"Silva, Junior Carlos"`` — 1.4 read title ``Junior``, first + ``Carlos``. +- capitalizing ``"JOHN ST CLAIR"``, ``"JUAN DELA CRUZ"`` and ``"JOHN + MACHADO"`` — 1.4 wrote ``st``, ``dela`` and ``MacHado``; saint and + rank particles keep their capital, and a name opening Mac is + title-cased unless the vocabulary records its casing (``MacDonald``). + +.. doctest:: + + >>> parse("Otto Graf Lambsdorff").family + 'Graf Lambsdorff' + >>> parse("Juan Sta. Maria Reyes").family, parse("Juan Delos Reyes").family + ('Sta. Maria Reyes', 'Delos Reyes') + >>> parse("María del Carmen García").middle + 'del Carmen' + >>> parse("طارق عبد العزيز").family + 'عبد العزيز' + >>> parse("Ortega y Gasset VI").family + 'Gasset' + >>> parse("John Smith (PhD, MBA)").suffix, parse("Justin (JP) Stearns").nickname + ('PhD, MBA', 'JP') + >>> parse("Alexandre Dumas père").suffix + 'père' + >>> parse("Jean Dupont, Pr").title, parse("Juan Pérez, Profesor").title + ('Pr', 'Profesor') + >>> parse("Jane Doe KC").suffix, parse("Jane Doe MSW").suffix + ('KC', 'MSW') + >>> parse("John⁣Smith").family + 'Smith' + >>> [parse(n).given for n in ("Maj Sjöwall", "Gen Goto", "Pharaoh Mwale", + ... "Saint Clair Soares")] + ['Maj', 'Gen', 'Pharaoh', 'Saint'] + >>> parse("Brig Gen Mark Kimmitt").title + 'Brig Gen' + >>> parse("Silva, Junior Carlos").given + 'Junior' + >>> [str(parse(n).capitalized()) for n in ("JOHN ST CLAIR", + ... "JUAN DELA CRUZ", "JOHN MACHADO", "RONALD MACDONALD")] + ['John St Clair', 'Juan Dela Cruz', 'John Machado', 'Ronald MacDonald'] diff --git a/docs/modules.rst b/docs/modules.rst index 08ecfb4ec..87a4d045f 100644 --- a/docs/modules.rst +++ b/docs/modules.rst @@ -1,8 +1,8 @@ API reference ============= -The 2.0 API ------------ +Classes and functions +--------------------- Parsing ~~~~~~~ @@ -59,7 +59,7 @@ Results — within one merged piece the tag is role-blind and every view joins the pair with a space, so it renders "Ph. D." as one piece in whatever role it lands — ``Ph. D., Jr.`` gives - ``last_list`` ``['Ph. D.']`` — and since #436 it also spans the pieces of one + family ``'Ph. D.'`` — and since #436 it also spans the pieces of one SUFFIX entry, the run of post-nominals the writer wrote without a comma — "MD PhD" — which is the half the suffix view alone reads and joins with a space instead of ", "). Every other tag is namespaced diff --git a/docs/usage.rst b/docs/usage.rst index ea2ad9c2f..040685eb6 100644 --- a/docs/usage.rst +++ b/docs/usage.rst @@ -51,7 +51,8 @@ starts a run of post-nominals and titles (``John Smith PhD Prof`` reads title ``Prof``; see :ref:`trailing-titles`), or, at the end of the given name in form 2, it is a listed salutation such as ``Mrs`` (``Doe, Jane Mrs`` reads title ``Mrs`` since 2.6; -``Lexicon.after_given_titles`` lists them). +``Lexicon.default().words(kind="salutation", after_given=True)`` lists +them). The last two differ in what the comma is doing. In form 2 it separates the family name from the rest, so the family name comes first; in form @@ -960,8 +961,9 @@ a nickname however it is written: >>> parse("JEFFREY (JD) BRICKEN").nickname 'JD' -``JD`` is in ``suffix_acronyms_ambiguous``; see -:ref:`ambiguous-words` for what that field marks and how to add to it. +``JD`` holds a ``given`` reading beside its credential one +(``Lexicon.default().readings_of("jd")``); see :ref:`ambiguous-words` +for what that second reading marks and how to add one. :ref:`brackets` covers the full order these readings are tried in, and how to configure the delimiter pairs. @@ -1017,7 +1019,8 @@ The rule is bounded so it doesn't swallow ordinary names: - **A given name written short** is that given name (since 2.7): the traditional English abbreviations such as ``Chas.``, ``Geo.`` and ``Wm.``, and the Filipino ``Ma.`` (María) and ``Jo.`` (José). - ``Lexicon.given_name_abbreviations`` lists them; written bare, or + Each is a ``given`` reading carrying ``same_as``, the full name + (``Lexicon.default().readings_of("chas")``); written bare, or anywhere else in the name, each word reads as it would unlisted. .. doctest:: @@ -1035,8 +1038,8 @@ The rule is bounded so it doesn't swallow ordinary names: >>> parse("Xyz. Smith, John").family 'Xyz. Smith' -Because this is structural rather than vocabulary-driven, emptying -``titles`` does not switch it off; see :ref:`title-detection-off`. +Because this is structural rather than vocabulary-driven, taking out +every title reading does not switch it off; see :ref:`title-detection-off`. .. _trailing-titles: @@ -1102,8 +1105,8 @@ When the parser had to guess Some names have no single correct reading. A leading ``Van`` could be a given name — it really is for the actor Van Johnson — or the start of a family name, as it is for President Van Buren. Both are the same -shape, so no rule can tell them apart. 2.0 takes the more likely -reading and *records* the choice on ``ambiguities`` rather than +shape, so no rule can tell them apart. The parser takes the more +likely reading and *records* the choice on ``ambiguities`` rather than deciding silently: .. doctest:: diff --git a/nameparser/AGENTS.md b/nameparser/AGENTS.md index 7b67f0349..82a41a006 100644 --- a/nameparser/AGENTS.md +++ b/nameparser/AGENTS.md @@ -82,7 +82,7 @@ Python's `\w` and `isalpha()` do not match Mn, so a letter written as base plus ### Widths and the initial shape -A width test must count letters, not code points (`composed_len`): admitting a decomposed word to the caps set put `John Smith, MÁ` on the pair exception's wrong side, and the review's grid found it where the rows (all three letters long) did not — put a two-letter word in every caps grid. The initial shape (`_INITIAL`, `\w\.`, v1-verbatim and public) is not widened but asked of the COMPOSED word, in `is_initial_shaped` and in `_render`'s two copies; and the single-letter connective tests count composed letters, since a decomposed `й` is two code points (#665, where `É. y É. Smith` and `John й Smith` read differently decomposed). +A width test must count letters, not code points (`composed_len`): admitting a decomposed word to the caps set put `John Smith, MÁ` on the pair exception's wrong side, and the review's grid found it where the rows (all three letters long) did not — put a two-letter word in every caps grid. The initial shape (`_INITIAL`, `\w\.`, v1-verbatim, and public as `REGEXES['initial']` until 3.0 removed it, #1008) is not widened but asked of the COMPOSED word, in `is_initial_shaped` and in `_render`'s two copies; and the single-letter connective tests count composed letters, since a decomposed `й` is two code points (#665, where `É. y É. Smith` and `John й Smith` read differently decomposed). ### Compose the WHOLE predicate diff --git a/nameparser/_lexicon.py b/nameparser/_lexicon.py index d0675361c..7b1aa80a0 100644 --- a/nameparser/_lexicon.py +++ b/nameparser/_lexicon.py @@ -1331,6 +1331,7 @@ def _target_readings(word: str, member: frozenset[str], heads: frozenset[str] | None, on: dict[str, dict[str, Any]], hints: dict[str, dict[str, Any]], + written_for: str | None = None, ) -> dict[str, dict[str, Any]]: """The readings, kind -> properties, that compile `word` into the sets `member` names, keeping each reading `word` holds (`on`) where @@ -1424,7 +1425,7 @@ def props(kind: str) -> dict[str, Any]: names["given"] = props("given") if "given" in names: if "given_name_abbreviations" in member: - names["given"].setdefault("same_as", word) + names["given"].setdefault("same_as", written_for or word) else: names["given"].pop("same_as", None) if (member & {"suffix_acronyms_ambiguous", "suffix_words_ambiguous"} @@ -1508,7 +1509,8 @@ def _inert_dropped(sets: Mapping[str, frozenset[str]], def _edits_toward(pool: Pool, current: Mapping[str, frozenset[str]], - target: Mapping[str, frozenset[str]]) -> tuple[_Edit, ...]: + target: Mapping[str, frozenset[str]], *, + keep_held: bool = False) -> tuple[_Edit, ...]: """The edits that take the readings of `pool`, which compile to the sets `current`, toward the sets `target` (by `_LexiconSets` name; a set it leaves out stays as `current` has it), word by word: for @@ -1518,7 +1520,11 @@ def _edits_toward(pool: Pool, current: Mapping[str, frozenset[str]], the refusals of a 2.x spelling and the load of a 2.x pickle share; a best effort, so each caller compiles the result where it must be exact. A new kind is taken from a reading the word holds off where - it holds one.""" + it holds one. With `keep_held`, a kind one of whose readings already + holds the properties wanted is left alone, so a call need not + rewrite another file's reading of it (`ma`'s Filipino `same_as`); + the set a property marks holds the word if any reading carries it, + so this is right wherever the result compiles exact.""" ri = _index_module() fields = _SET_FIELDS full = _inert_dropped({f: target.get(f, current[f]) for f in fields}) @@ -1545,8 +1551,13 @@ def heads(phrases: frozenset[str], word: str) -> frozenset[str]: if f != "title_phrases" and w in full[f]) new_heads = heads(full["title_phrases"], w) moved = new_heads != heads(current["title_phrases"], w) + # the full name a given name written short stands for, which no + # set records: kept from the reading that holds it (`ma`'s + # Filipino `maría`), the word itself where none does + written_for = next((p["same_as"] for p in _props_by_kind(held) + .get("given", ()) if "same_as" in p), w) want = _target_readings(w, member, new_heads if moved else None, - on, every) + on, every, written_for) dropped = [k for k in on if k not in want] gone_categories = {k for k in dropped if ri.CATEGORY[k] == k} for k in dropped: @@ -1562,7 +1573,8 @@ def heads(phrases: frozenset[str], word: str) -> frozenset[str]: for k, p in sorted(want.items(), key=lambda kp: ri.CATEGORY[kp[0]] != "name"): if k in after: - if k in on and all(q == p for q in after[k]): + if k in on and (any if keep_held else all)( + q == p for q in after[k]): continue delta = {x: v for x, v in p.items() if any(q.get(x) != v for q in after[k])} @@ -1642,14 +1654,22 @@ def _translation(lexicon: Lexicon, changes: Mapping[str, frozenset[str]], wanted = {f: target.get(f, current[f]) for f in _SET_FIELDS} wanted["titles_ambiguous"] = wanted["titles_ambiguous"] - wanted["titles"] wanted = _inert_dropped(wanted) - try: - edits = _edits_toward(lexicon._pool(), current, target) - result = _inert_dropped( - _built(lexicon._base, lexicon._edits + edits, lexicon._chosen, - lexicon._strict)[1]) - exact = all(result[f] == wanted[f] for f in _SET_FIELDS) - except (TypeError, ValueError): - edits, exact = (), False + def attempt(keep_held: bool) -> tuple[tuple[_Edit, ...], bool]: + try: + edits = _edits_toward(lexicon._pool(), current, target, + keep_held=keep_held) + result = _inert_dropped( + _built(lexicon._base, lexicon._edits + edits, + lexicon._chosen, lexicon._strict)[1]) + except (TypeError, ValueError): + return (), False + return edits, all(result[f] == wanted[f] for f in _SET_FIELDS) + + # the fewer calls first, keeping a reading another file holds of a + # kind; every reading of the kind rewritten where that is not exact + edits, exact = attempt(True) + if not exact: + edits, exact = attempt(False) if not edits and exact: return ("the lexicon already reads those words so, an entry it " "would add or remove being a marker without its word's " @@ -1708,9 +1728,22 @@ def _refusal(where: str, lexicon: Lexicon, f"remove(*words, kind=...); for these words, " + _translation(lexicon, { f: cast("frozenset[str]", v) for f, v in removed.items()}, - # a bare constructor call names the base it starts from - "Lexicon.empty()" if where.startswith("Lexicon()") - and lexicon == Lexicon.empty() else "lexicon")) + _receiver(where, lexicon))) + + +def _receiver(where: str, lexicon: Lexicon) -> str: + """What an offered call is made on: a bare constructor call names + the lexicon it built, which the caller holds in no variable, and + any other spelling the `lexicon` it was called on.""" + if (not where.startswith("Lexicon()") or lexicon._base != "empty" + or lexicon._edits or lexicon._languages): + return "lexicon" + lists = [f"{name}={{{', '.join(map(repr, sorted(getattr(lexicon, name))))}}}" + for name in _LIST_FIELDS if getattr(lexicon, name)] + if lexicon.capitalization_exceptions: + lists.append("capitalization_exceptions=" + f"{lexicon.capitalization_exceptions!r}") + return f"Lexicon({', '.join(lists)})" if lists else "Lexicon.empty()" _PICKLE_2X_DEPRECATION = ( @@ -1805,7 +1838,8 @@ class LexiconReading: shipped files holding it, sorted, empty for a reading a caller added. Whether the parse READ a word this way is the parse's question, :attr:`Token.reading`: a title reading beside a ``given`` - one is a title only inside a title phrase (rules/H9).""" + one is a title only inside a title phrase (rules/H9). Added in + 3.0.""" text: str kind: str @@ -1898,7 +1932,7 @@ class Lexicon: #: written "jose" matches "José" too. No other rule reads it. Empty #: by default, so the switch reads as without it; a caller passes #: a list of its own, and ``locales.ES`` carries a Spanish list, with - #: the switch on. + #: the switch on. Added in 3.0. given_names: frozenset[str] = frozenset() #: Lowercase word -> case mask used by capitalized(): the word's #: own letters and digits, each in the case it takes ("phd" -> @@ -1991,10 +2025,25 @@ def __init__( @classmethod def empty(cls) -> Lexicon: + """A lexicon holding no readings, no surnames, no given names + and no case masks, equal to ``Lexicon()``: the start for a + fragment to join onto another with ``|``, or for a vocabulary + built from nothing. A parser over it still reads by shape and + position (``Dr. John Smith`` keeps title ``Dr.``, a period-marked + word opening a name), and, the default :class:`Policy + ` dividing unspaced Hangul, warns at + construction that no surnames cover it.""" return cls() @classmethod def default(cls) -> Lexicon: + """The shipped vocabulary, the lexicon ``Parser()`` uses when + given none: every reading the language files turn on by + default, the Korean census surnames and the case masks, with no + :attr:`given_names` and no choice of languages. Cached, so + every call returns the same object, which being immutable is + safe to share; configure a copy with :meth:`add`, + :meth:`remove` or ``|``.""" return _default_lexicon() # -- dunders ---------------------------------------------------------- @@ -2168,14 +2217,14 @@ def languages(self) -> frozenset[str]: with its packs' (decisions/vocabulary-model, 2026-11-02 #754, item 4): the shipped readings are filtered by the choice, and the lexicon's own :meth:`add` and :meth:`remove` calls apply - after it.""" + after it. Added in 3.0.""" return self._languages @property def strict(self) -> bool: """Whether the choice is strict: only the chosen languages' vocabulary files, mul.toml and native-script vocabulary are - read.""" + read. Added in 3.0.""" return self._strict # -- access ----------------------------------------------------------- @@ -2187,7 +2236,7 @@ def readings_of(self, word: str) -> tuple[LexiconReading, ...]: of. `word` is normalized as the lexicon stores it, a phrase (``"z domu"``) whole. "Is X a title here?" is ``any(r.category == "title" for r in lexicon.readings_of(x))``; whether a parse - read it as one is :attr:`Token.reading`'s.""" + read it as one is :attr:`Token.reading`'s. Added in 3.0.""" _reject_buffer(word, "readings_of()", "characters") if not isinstance(word, str): raise TypeError(f"readings_of() takes a word, a string, got " @@ -2222,7 +2271,7 @@ def words(self, *, kind: str, **properties: object) -> tuple[str, ...]: false, alike) (``words(kind="title", addresses="given")``, the titles that address a person by the given name). A phrase the lexicon holds - whole (``"z domu"``) is one entry.""" + whole (``"z domu"``) is one entry. Added in 3.0.""" k = _check_kind(kind, "words") assert k is not None ri = _index_module() diff --git a/nameparser/_locale.py b/nameparser/_locale.py index 765da52f9..c1a2a8212 100644 --- a/nameparser/_locale.py +++ b/nameparser/_locale.py @@ -39,8 +39,9 @@ @dataclass(frozen=True, slots=True) class Locale: """A named, shareable bundle of vocabulary and behavior for a - naming tradition: a lexicon fragment plus a policy patch, applied - together by :func:`nameparser.parser_for`. The packs shipped with + naming tradition: a choice of languages, a lexicon fragment and a + policy patch, applied together by :func:`nameparser.parser_for`. + The packs shipped with nameparser live in :mod:`nameparser.locales`; building your own needs no registration -- construct one and pass it to ``parser_for``. See the :doc:`locale packs guide ` for @@ -65,7 +66,8 @@ class Locale: #: drops other languages' vocabulary readings. A tag selects its own #: vocabulary file and its ancestors' (``pt-BR`` selects ``pt`` #: too), never its descendants'. Empty, the default, chooses - #: nothing, and the default vocabulary stands as it is. + #: nothing, and the default vocabulary stands as it is. Added in + #: 3.0. languages: frozenset[str] = field( default=frozenset(), metadata={ADDED_IN: "3.0.0"}) diff --git a/nameparser/_parser.py b/nameparser/_parser.py index dafba4fb2..895f3a21a 100644 --- a/nameparser/_parser.py +++ b/nameparser/_parser.py @@ -145,13 +145,14 @@ def __repr__(self) -> str: def languages(self) -> frozenset[str]: """The languages this parser's vocabulary is chosen under, :attr:`Lexicon.languages` read through: set by - :func:`parser_for`, empty where none was chosen.""" + :func:`parser_for`, empty where none was chosen. Added in + 3.0.""" return self.lexicon.languages @property def strict(self) -> bool: """Whether the choice is strict, :attr:`Lexicon.strict` read - through.""" + through. Added in 3.0.""" return self.lexicon.strict def parse(self, text: str) -> ParsedName: diff --git a/nameparser/_pipeline/_assemble.py b/nameparser/_pipeline/_assemble.py index b631d181b..e92eeaae1 100644 --- a/nameparser/_pipeline/_assemble.py +++ b/nameparser/_pipeline/_assemble.py @@ -4,11 +4,14 @@ token's reading), dropped, order and structure (for ParsedName.order), ambiguities (by index; one carrying ``field_tail`` is worded from its word's final role, -#626), and segments, for the part a withdrawn connective report's -letter ends (rules/P8, #721). +#626), segments, for the part a withdrawn connective report's +letter ends (rules/P8, #721), and comma_offsets, for the comma part +rules/R9's Turkic arm keeps a marker and its father's name in. Produces: a validated ParsedName, each token carrying the reading the parse took (`Token.reading`), a word of the family's second unit -marked `Reading.second_surname` where O6's tag says so (rules/R10) -- +marked `Reading.second_surname` where O6's tag says so (rules/R10) +and `Reading.first_surname` where its first-surname tag does +(rules/R11) -- the constructor re-checks every invariant (span order/bounds, ambiguity subset), so a pipeline bug that would produce an invalid result dies HERE, not in a renderer @@ -19,7 +22,9 @@ an ending rule reads as a patronymic (rules/R9); and Lexicon.surnames, suffix_acronyms and suffix_acronyms_ambiguous, for the source of a token's reading where no tag gives it; and the Lexicon's readings by word (Lexicon._files), -for Reading.languages, the files whose reading is on in it. +for Reading.languages, the files whose reading is on in it, with +every vocabulary field (`_lexicon._VOCAB_FIELDS`, `_listed_whole`) +for whether the lexicon lists a word whole rather than by its parts. Structural tokens (dropped maiden markers) are omitted, like delimiter characters. A main-stream token that somehow reaches here with no role diff --git a/nameparser/_pipeline/_assign.py b/nameparser/_pipeline/_assign.py index 40574a55f..25653a58a 100644 --- a/nameparser/_pipeline/_assign.py +++ b/nameparser/_pipeline/_assign.py @@ -4,11 +4,13 @@ one_case, and tail_reads -- rules/S2's trailing run as group read it once, after P3's joins and before the particle chain (#614), which the main segment's placement takes rather than peeling again, save where -group's joins left no name word in front of it. +group's joins left no name word in front of it; and interpunct_offsets, +whose 间隔号 keeps a transcription in its declared order (rules/W4). Produces: tokens with roles set on every main-stream token, and the folded-middle and attached tags on the particles P6's attachment joins to the family after a family comma (#613; the attached tag, 3.0's 4c, -tells rules/R9 the attachment from O3's fold). +tells rules/R9 the attachment from O3's fold); and order, the order +the positional read used, None where none happened (ParseState.order). Reads: Policy.name_order (#270), is_suffix_lenient on the trailing piece of a two-part comma name, and Policy.script_orders (#271, which overrides it when every name piece is written wholly in one script, or @@ -16,7 +18,9 @@ across pieces); token/piece tags; Lexicon only through tags already applied by classify (plus the leading-title period rule), and Lexicon.given_name_titles, which an ambiguity's alternative asks for -rules/P5's given-name-title licence. +rules/P5's given-name-title licence; and Lexicon.title_phrases, titles +and titles_ambiguous through _vocab.in_a_title_phrase, for rules/H9's +TITLE_OR_NAME report on a contested word opening the name. Implements rules H2, H4, H5, H6, H7, H8, N3, O4, O5, S10, W4, and P6 after a family comma, of docs/design/rules/, diff --git a/nameparser/_pipeline/_classify.py b/nameparser/_pipeline/_classify.py index 5d7f40b2c..4e66f6843 100644 --- a/nameparser/_pipeline/_classify.py +++ b/nameparser/_pipeline/_classify.py @@ -1,6 +1,8 @@ """Stage: classify. -Consumes: tokens, comma_offsets (with token roles, the two halves of +Consumes: tokens, segments (the part a title phrase or a split +credential's pair is read within), extracted (whether a delimited +clause holds a word with a comma on it, rules/S1), comma_offsets (with token roles, the two halves of the structural-boundary test the marker pass applies -- see _vocab.tag_marker_runs), and one_case where an earlier stage recorded it -- segment writes it lazily where C2's flag on a part past the @@ -12,10 +14,13 @@ recorded for the later stages that read it (#289/#516) and left alone where an earlier stage already asked. Reads: every Lexicon vocabulary field except surnames and -honorific_tails, which script_segment consumes upstream, and the kinds +honorific_tails, which script_segment consumes upstream, and +given_names, which only post_rules reads (rules/O6); and the kinds of its suffix words and particles (Lexicon's generation_words, credential_words, honorific_words and patronymic_particles, internal sets, 3.0) and the +suffix words whose Title case leans to the name +(title_case_name_words, rules/S6, #1039) and the titles written short (title_abbreviations, #999) and the names a community gives every member (shared_names) and the given names written behind a particle (particle_given_names, #704) and the diff --git a/nameparser/_pipeline/_post_rules.py b/nameparser/_pipeline/_post_rules.py index 7a1f3e2b1..31ac3599f 100644 --- a/nameparser/_pipeline/_post_rules.py +++ b/nameparser/_pipeline/_post_rules.py @@ -7,7 +7,10 @@ below reads to find the separators the writer typed (#436/#437), and extracted, one_case and original, which S1's set-apart pass reads, settling S13's members with its own (#810, #792), original also read -by R1's entry pass for a comma a set-apart clause kept. +by R1's entry pass for a comma a set-apart clause kept; and order, +the order assign read the name under, which H1, P1's fold, P6's +attachment, P10, O1, O2, O6 and O7 key on rather than +Policy.name_order (#395). Produces: tokens with roles adjusted by the post rules (among them the TITLE and SUFFIX roles of a set-apart clause's words, rules/S1, each word's comma trimmed off its text and span, and the diff --git a/nameparser/_pipeline/_script_segment.py b/nameparser/_pipeline/_script_segment.py index b2101725e..94b618faa 100644 --- a/nameparser/_pipeline/_script_segment.py +++ b/nameparser/_pipeline/_script_segment.py @@ -3,7 +3,8 @@ Consumes: tokens, segments, structure, interpunct_offsets, segmenter, one_case where segment recorded it, and comma_offsets, for the own-words span `_one_case` asks the fact over where segment did not -(#613; the answer is not recorded, classify recording its own). +(#613; the answer is not recorded, classify recording its own), and +original, for the peel's ASCII bail. Produces: tokens, by two independent splits into sub-slices -- a listed honorific peeled off the END of the name's last non-post-nominal token, in whichever of the name's runs that falls diff --git a/nameparser/_pipeline/_state.py b/nameparser/_pipeline/_state.py index 09dcc01fd..76d8fce97 100644 --- a/nameparser/_pipeline/_state.py +++ b/nameparser/_pipeline/_state.py @@ -368,11 +368,14 @@ class ParseState: assign -> the remaining token roles AND `order`, the effective order it read them under, AND the folded-middle and attached tags on the particles P6's attachment joins after a family comma (#613); - post_rules -> roles again (a delimited member S13 settles becomes - the suffix, #810), the folded-middle tag on what O3 folds and the + post_rules -> roles again (the TITLE and SUFFIX roles of a clause + S1 sets apart, each word's comma trimmed off its text and span, + #792; a delimited member S13 settles becomes the suffix, #810), + the folded-middle tag on what O3 folds and the no-comma attachment joins, with the attached tag on the latter, the second-surname tag on the words of the family's second unit - where O6 reads two (rules/R10), and the ambiguity P6's no-comma + where O6 reads two (rules/R10), the first-surname tag on the + words of its first unit (rules/R11), and the ambiguity P6's no-comma attachment reports (#467) and SUFFIX_OR_NICKNAME on a member it leaves a nickname. diff --git a/nameparser/_policy.py b/nameparser/_policy.py index de587304b..914ede531 100644 --- a/nameparser/_policy.py +++ b/nameparser/_policy.py @@ -742,7 +742,8 @@ class Policy: #: before middle_as_family, which then folds whatever middles are #: left. A name whose second given word is a middle name gains it #: in the given name ("John Quincy Adams" gives given ``John - #: Quincy``), so it is off by default and reports nothing. + #: Quincy``), so it is off by default and reports nothing. Added in + #: 3.0. two_given_names: bool = field( default=False, metadata={ADDED_IN: "3.0.0"}) @@ -954,6 +955,8 @@ class PolicyPatch: unlisted_dotted_suffixes: bool | _Unset = UNSET unlisted_caps_suffixes: CapsSuffixes | _Unset = UNSET two_family_names: bool | _Unset = UNSET + #: Patches :attr:`Policy.two_given_names`, a scalar (later wins). + #: Added in 3.0. two_given_names: bool | _Unset = field( default=UNSET, metadata={ADDED_IN: "3.0.0"}) diff --git a/nameparser/_types.py b/nameparser/_types.py index 2995395c6..8abf17603 100644 --- a/nameparser/_types.py +++ b/nameparser/_types.py @@ -403,7 +403,7 @@ class Token: #: one :meth:`ParsedName.replace` or :meth:`Parser.revise` builds, #: or one loaded from a pickle written before 3.0. Not compared: #: two tokens equal without it, as before 3.0, and - #: :meth:`ParsedName.comparison_key` ignores it. + #: :meth:`ParsedName.comparison_key` ignores it. Added in 3.0. reading: Reading | None = field( default=None, compare=False, metadata={ADDED_IN: "3.0.0"}) @@ -597,8 +597,8 @@ class AmbiguityKind(StrEnum): certainty.""" #: Reserved: the name's field order itself is uncertain (e.g. a - #: two-word name under a non-default name_order). Not yet emitted; - #: planned for 2.x. A lone name word is GIVEN_OR_FAMILY's, not + #: two-word name under a non-default name_order). Not yet + #: emitted. A lone name word is GIVEN_OR_FAMILY's, not #: this. ORDER = "order" #: Delimited content is an ambiguous suffix acronym, so it reads @@ -762,8 +762,9 @@ class AmbiguityKind(StrEnum): #: A title that may also be a given name ("Dean", rules/H9) reports #: it too, where it opens the name outside a title phrase and no #: comma has marked the given name: "Dean Smith" gives given "Dean". - #: Only a word some phrase of ``Lexicon.title_phrases`` holds - #: reports, so "Grace Smith" does not. + #: Only a word some title phrase holds (a pair a title reading's + #: ``attaches`` property licenses) reports, so "Grace Smith" does + #: not. TITLE_OR_NAME = "title-or-name" #: An ambiguous particle is either a particle or a name in its own #: right -- "Van Johnson" is the actor's given name, a bare @@ -798,7 +799,8 @@ class AmbiguityKind(StrEnum): #: an INITIAL and this reports the fork: "jose e maria santos" gives #: middle "e maria" and "JOSE E MARIA SANTOS" middle "E MARIA", #: both flagged. Only a letter the vocabulary marks both ways - #: reports -- ``Lexicon.conjunctions_ambiguous``, "e" and "i" by + #: reports -- a conjunction reading written ``may_be_initial`` + #: (``Lexicon.words(kind="conjunction", may_be_initial=True)``), "e" and "i" by #: default -- because a letter outside it is not in doubt: "JUAN GARCIA Y #: LOPEZ" joins into family "GARCIA Y LOPEZ" and reports nothing, #: as does the Cyrillic "ХОСЕ И МАРИЯ САНТОС" -- whose join lands @@ -985,7 +987,8 @@ class Reading: #: A suffix reading is matched by the word's suffix entry whatever #: its finer kind, a word having one. An ``Ambiguity.alternative`` #: names the files holding it for any word the report names. Not - #: compared: two readings are equal without it, as in 2.x. + #: compared: two readings are equal without it, as in 2.x. Added + #: in 3.0. languages: tuple[str, ...] = field( default=(), compare=False, metadata={ADDED_IN: "3.0.0"}) #: True for a word of the family's second name unit, where a rule @@ -1242,7 +1245,7 @@ class ParsedName: #: (a name built by hand, or a 2.x pickle) renders as 2.x did: #: given-first, its words joined by spaces. #: Left out of ``==`` and :meth:`comparison_key`, so two equal - #: names can render differently. + #: names can render differently. Added in 3.0. order: tuple[Role, Role, Role] | None = field( default=None, compare=False, metadata={ADDED_IN: "3.0.0"}) @@ -1376,14 +1379,21 @@ def _text_for(self, *roles: Role, tag: str | None = None, @property def title(self) -> str: + """The words read as titles ("Dr.", "Sir"), wherever in the + name they stand, space-joined in written order; ``''`` for a + name with none.""" return self._text_for(Role.TITLE) @property def given(self) -> str: + """The given name's words, space-joined in written order; + ``''`` for a name with none ("de la Vega").""" return self._text_for(Role.GIVEN) @property def middle(self) -> str: + """The middle names' words, space-joined in written order; + ``''`` for a name with none.""" return self._text_for(Role.MIDDLE) @property @@ -1392,18 +1402,37 @@ def middle(self) -> str: # folded family words" (O3's fold and P6's tussenvoegsel, which # render before the rest of the family) def family(self) -> str: + """The family name's words, particles included ("de la + Vega"), space-joined in written order, save the middle names + ``middle_as_family`` folds in, which come first; ``''`` for a + name with none ("Williams" is a given name alone).""" return self._text_for(Role.FAMILY) @property def suffix(self) -> str: + """The words after the name ("Jr.", "PhD", "さん"): each entry + the writer set apart joined to the next with ``", "``, the + words of one entry with a space ("MD PhD, Jr."); ``''`` for a + name with none. :attr:`generation`, :attr:`credentials` and + :attr:`honorifics` divide it by kind.""" return self._text_for(Role.SUFFIX) @property def nickname(self) -> str: + """The words written inside nickname delimiters (quotes, + parentheses; :attr:`Policy.nickname_delimiters + `), space-joined in + written order, the delimiters dropped; ``''`` for a name with + none.""" return self._text_for(Role.NICKNAME) @property def maiden(self) -> str: + """The maiden name's words, those a birth marker ("née", + "geb.") introduces or a pair in :attr:`Policy.maiden_delimiters + ` encloses, space-joined + in written order, the marker dropped; ``''`` for a name with + none.""" return self._text_for(Role.MAIDEN) # -- derived views (filters over roles, STABLE tags and readings) --- @@ -1413,16 +1442,28 @@ def maiden(self) -> str: # base (the family without its particles) and the particles # themselves" def family_particles(self) -> str: + """The particles of :attr:`family` ("de la" in "Juan de la + Vega"), space-joined as :attr:`family` writes them; ``''`` for + a family with none. A family that :meth:`replace` or + Parser.revise wrote carries no particle tags, so its words are + all in :attr:`family_base`.""" return self._text_for(Role.FAMILY, tag="particle", unless_tag=UNJOINED_TAG) @property def family_base(self) -> str: + """:attr:`family` without its particles ("Vega" in "Juan de la + Vega"), space-joined as :attr:`family` writes them, so the two + views together hold the family's words; ``''`` for a name with + no family.""" return self._text_for(Role.FAMILY, without_tag="particle", unless_tag=UNJOINED_TAG) @property def surnames(self) -> str: + """The middle names and the family name together ("Q. Xavier + de la Vega" in "Juan Q. Xavier de la Vega"), space-joined in + written order; ``''`` for a name with neither ("Williams").""" return self._text_for(Role.MIDDLE, Role.FAMILY) @property @@ -1431,7 +1472,7 @@ def surnames(self) -> str: def generation(self) -> str: """The suffix words read as a generation or an agnomen ("Jr.", "III", "Filho"), joined as :attr:`suffix` joins them; ``''`` - for a name with none.""" + for a name with none. Added in 3.0.""" return self._suffix_view(_GENERATION_KINDS) @property @@ -1441,7 +1482,7 @@ def credentials(self) -> str: "Esq.", "Rinpoche"), joined as :attr:`suffix` joins them; ``''`` for a name with none. A suffix word with no :attr:`Token.reading` (one :meth:`replace` or Parser.revise - wrote) is counted here.""" + wrote) is counted here. Added in 3.0.""" return self._suffix_view(None) @property @@ -1453,6 +1494,10 @@ def honorifics(self) -> str: @property def given_names(self) -> str: + """The given name and the middle names together ("Juan Q. + Xavier" in "Juan Q. Xavier de la Vega"), space-joined in + written order; ``''`` for a name with neither ("de la + Vega").""" return self._text_for(Role.GIVEN, Role.MIDDLE) @property @@ -1464,7 +1509,7 @@ def patronymic(self) -> str: Slavic rule), in whichever field the parse put them, which keeps them; ``''`` for a name with none. Space-joined in written order, the family's words in the order :attr:`family` - writes them.""" + writes them. Added in 3.0.""" parts: list[str | list[str]] = [] family: list[str] | None = None folded: list[str] = [] @@ -1499,7 +1544,7 @@ def surname2(self) -> str: and a hyphenated one included, for a name unpickled from 2.x, and once :meth:`replace` or Parser.revise rewrites ``family``, whose words then carry no reading; rewriting another field - keeps it.""" + keeps it. Added in 3.0.""" # the parse marks family words only; the role test keeps a # hand-built token marked in another field out of the view return " ".join( @@ -1524,8 +1569,9 @@ def surname1(self) -> str: ``two_family_names``, ``de la Vega`` in ``Juan de la Vega del Valle``), :attr:`surname2` being the other; elsewhere the whole :attr:`family`. A middle name ``middle_as_family`` folds into - the family is in neither (``García López`` in ``Ana María García - López`` under ``middle_as_family``).""" + the family is in neither (``María García`` in ``Ana María García + López`` under ``middle_as_family``, whose surname1 is + ``López``). Added in 3.0.""" return self._text_for(Role.FAMILY, first_unit=self._first_unit()) @property @@ -1533,7 +1579,7 @@ def surname1_particles(self) -> str: """:attr:`surname1`'s particles, as :attr:`family_particles` reads the whole family's (``de la`` in ``Juan de la Vega del Valle`` under ``two_family_names``), so the two units' particles are the - family's.""" + family's. Added in 3.0.""" return self._text_for(Role.FAMILY, tag="particle", unless_tag=UNJOINED_TAG, first_unit=self._first_unit()) @@ -1543,7 +1589,7 @@ def surname1_base(self) -> str: """:attr:`surname1` without its particles, as :attr:`family_base` reads the whole family (``Pérez`` in ``Juan Pérez de la Vega`` under ``two_family_names``, where :attr:`family_base` is - ``Pérez Vega``).""" + ``Pérez Vega``). Added in 3.0.""" return self._text_for(Role.FAMILY, without_tag="particle", unless_tag=UNJOINED_TAG, first_unit=self._first_unit()) @@ -1593,6 +1639,12 @@ def tokens_for(self, role: Role | str) -> tuple[Token, ...]: return tuple(t for t in self.tokens if t.role is role) def as_dict(self, *, include_empty: bool = True) -> dict[str, str]: + """The seven fields as a new dict keyed by each :class:`Role`'s + value (``"title"``, ``"given"``, ...), in :class:`Role` order, + each value the string its property gives, ``''`` included; + ``include_empty=False`` leaves the empty fields out. The views + derived from the fields (:attr:`family_base`, + :attr:`generation`, ...) are not in it.""" # _text_for handles the suffix ", "-join (single-role SUFFIX call) d = {role.value: self._text_for(role) for role in Role} if not include_empty: diff --git a/nameparser/locales/__init__.py b/nameparser/locales/__init__.py index 4d76334ef..0980db5a6 100644 --- a/nameparser/locales/__init__.py +++ b/nameparser/locales/__init__.py @@ -8,7 +8,7 @@ first access and caches the Locale here. Layering: this package imports pack modules, which import _locale/ -_lexicon/_policy only (enforced by tests/v2/test_layering.py). +_lexicon/_policy/_types only (enforced by tests/v2/test_layering.py). """ from __future__ import annotations @@ -72,8 +72,10 @@ def available() -> tuple[str, ...]: def ja_segmenter(*, gbdt: bool = False) -> Segmenter: """The Japanese segmenter factory (#272), needing the optional - ``pip install nameparser[ja]``; see :mod:`nameparser.locales.ja` for - what it wraps, what it declines, and its licensing. + ``pip install 'nameparser[ja]'``; the docstring of + ``nameparser.locales.ja`` says what it wraps, what it declines, and + its licensing, and :ref:`segmenter-contract` what a segmenter owes + its caller. A real function rather than a _REGISTRY entry, because the registry resolves LOCALES: __getattr__ is annotated to return one, so routing diff --git a/nameparser/locales/ja.py b/nameparser/locales/ja.py index 1478c115a..b0c3358ee 100644 --- a/nameparser/locales/ja.py +++ b/nameparser/locales/ja.py @@ -157,7 +157,7 @@ def ja_segmenter(*, gbdt: bool = False) -> Segmenter: except ImportError as exc: raise ImportError( "the Japanese segmenter needs namedivider-python -- " - "install it with: pip install nameparser[ja]" + "install it with: pip install 'nameparser[ja]'" ) from exc divider = (namedivider.GBDTNameDivider() if gbdt else namedivider.BasicNameDivider()) diff --git a/nameparser/vocab/AGENTS.md b/nameparser/vocab/AGENTS.md index eef2d4bc7..10972df11 100644 --- a/nameparser/vocab/AGENTS.md +++ b/nameparser/vocab/AGENTS.md @@ -24,7 +24,7 @@ Record what the vetting found ON the word, not in a comment or a decisions sente ## A word that becomes a particle: its case -A particle reading is lowercased by case repair (rules/R4) unless its word carries a `cased` mask, so a word the default capitalizes as a name word loses its capital the moment a reading makes it a particle, in the default or under the pack that turns it on. #956 turned on Irish `nic`, `mhic` and `mic` with no mask, and `MÁIRE NIC GIOLLA PHÁDRAIG` repaired to `Máire nic Giolla Phádraig` under `ga`, where the default gives `Nic`; `ní` and `uí` had their masks and kept it. #1037 met it again in the default, giving Filipino `delos` a particle reading, and its first measurement, taken before the mask, read `JUAN DELOS REYES` repairing to `Juan delos Reyes`; `dela` had shipped that way, lowercased, where ORCID's Philippine family boxes write `Dela` three times in four (352 of 471, `dela` 60, `DELA` 59). Give a capitalized particle its `cased`, and test `capitalized()` both ways (repair and `force=True`) under the parser that reads it. +A particle reading is lowercased by case repair (rules/R4) unless its word carries a `cased` mask or the reading is of a kind written with a capital (`saint`, `rank`, #1069), so a word the default capitalizes as a name word loses its capital the moment a reading makes it a particle, in the default or under the pack that turns it on. #956 turned on Irish `nic`, `mhic` and `mic` with no mask, and `MÁIRE NIC GIOLLA PHÁDRAIG` repaired to `Máire nic Giolla Phádraig` under `ga`, where the default gives `Nic`; `ní` and `uí` had their masks and kept it. #1037 met it again in the default, giving Filipino `delos` a particle reading, and its first measurement, taken before the mask, read `JUAN DELOS REYES` repairing to `Juan delos Reyes`; `dela` had shipped that way, lowercased, where ORCID's Philippine family boxes write `Dela` three times in four (352 of 471, `dela` 60, `DELA` 59). Give a capitalized particle of any other kind its `cased`, and test `capitalized()` both ways (repair and `force=True`) under the parser that reads it. ## A suffix word that is also a name: its writing diff --git a/nameparser/vocab/README.md b/nameparser/vocab/README.md index 75f88a124..f0f2366e0 100644 --- a/nameparser/vocab/README.md +++ b/nameparser/vocab/README.md @@ -2,9 +2,9 @@ This folder holds the parser's default vocabulary: the titles, suffixes, particles, conjunctions, bound given names and maiden markers it recognizes, and the case of the words it capitalizes specially, written one word at a time. It is the place to edit when a word should be recognized, should stop being recognized, or should be recognized differently. -The parser never reads these files. `tools/vocab/build.py` compiles them into `nameparser/config/_compiled.py`, a committed module of sorted `frozenset` literals and one `dict` (`CAPITALIZATION_EXCEPTIONS`), which `Lexicon` reads; since 3.0 they are private (#1007), a caller listing a kind's words with `Lexicon.words(kind=...)`. Beside it the build writes a second committed module from the same readings, the reading index `nameparser/config/_index.py`, which keeps every reading with the tag of the file it is in and the properties the compiler reads, so that a choice of languages can compile from it later (3.0's phase 3), and a third, `nameparser/config/_rules.py`, holds the files' [rules](#rules) tables. So an edit here does nothing until you rebuild, and the tests fail, naming the command, when a committed module and the files disagree. +The parser never reads these files. `tools/vocab/build.py` compiles them into `nameparser/config/_compiled.py`, a committed module of sorted `frozenset` literals and one `dict` (`CAPITALIZATION_EXCEPTIONS`), which `Lexicon` reads; since 3.0 they are private (#1007), a caller listing a kind's words with `Lexicon.words(kind=...)`. Beside it the build writes a second committed module from the same readings, the reading index `nameparser/config/_index.py`, which keeps every reading with the tag of the file it is in and the properties the compiler reads, which a choice of languages (`Locale.languages`, 3.0's phase 3) compiles from, and a third, `nameparser/config/_rules.py`, holds the files' [rules](#rules) tables. So an edit here does nothing until you rebuild, and the tests fail, naming the command, when a committed module and the files disagree. -Why the model looks this way, and what each decision cost, is recorded in [decisions/vocabulary-model](../../docs/design/decisions/vocabulary-model.md). Where it is going (phases 2 to 4) is in [the vocabulary model RFC](../../docs/design/vocabulary-model-rfc.md). +Why the model looks this way, and what each decision cost, is recorded in [decisions/vocabulary-model](../../docs/design/decisions/vocabulary-model.md). The plan it was built to, phase by phase (phases 2 to 4, each section saying what has landed), is in [the vocabulary model RFC](../../docs/design/vocabulary-model-rfc.md). ## What a file means @@ -60,7 +60,7 @@ family = { us_census_2010 = 35053, wikidata = 315, as_of = 2026-10-19 } excluded = { nobility = "a common given name (Duke Kahanamoku), and a surname besides" } ``` -`duke` is deliberately not a title. It has no vocabulary reading at all, only its two name readings and an exclusion refusing the `nobility` reading with the reason. Anyone who adds `nobility = {}` to `duke` in any file gets a build error that quotes that reason. An exclusion means the reading is wrong wherever it is read. A reading that is right in its own language and wrong only as a default is written `default = false` instead ([Properties](#properties)), as `père` ("father", the French word for the Elder) is in `fr.toml`, so that choosing French turns it on; it was an exclusion until 3.0's phase 3c. Its son, `fils`, was an exclusion too until 2026-10-25, and is now on by default: its `generation` reading stands beside its `family` reading, which gates it by the name around it ([derived markers](#derived-markers)), so `Marie Fils` keeps her family name. +`duke` is deliberately not a title. It has no vocabulary reading at all, only its two name readings and an exclusion refusing the `nobility` reading with the reason. Anyone who adds `nobility = {}` to `duke` in any file gets a build error that quotes that reason. An exclusion means the reading is wrong wherever it is read. A reading that is right in its own language and wrong only as a default is written `default = false` instead ([Properties](#properties)), as `ml` ("mladší", the Younger, a pair of initials in other languages) is in `cs.toml`, so that choosing Czech turns it on. `père` ("father", the French word for the Elder) was an exclusion until 3.0's phase 3c and such a reading until 2026-11-03 (#684), and its son, `fils`, an exclusion until 2026-10-25; both are now on by default: each one's `generation` reading stands beside its `family` reading, which gates it by the name around it ([derived markers](#derived-markers)), so `Marie Fils` keeps her family name. ```toml [[word]] @@ -187,7 +187,7 @@ A property is **behavioral** when it compiles into a set and so changes how name | `completed_by` | behavioral | `bound_given` | none | `BOUND_PHRASES`: the words that complete the bound given name into one name unit past the first name word, a list (`completed_by = ["الله", "الرحمن"]` on `عبد`), each one word written folded: `سعيد عبد الله` reads family `عبد الله` ([rules/P12](../../docs/design/rules/P12-bound-given-name-and-the-word-that-completes-it.md), a bound given name and the word that completes it), while a word left off the list after it is a name of its own, as in `نوال عبد هاشم` (#694) | | `may_be_given` | stated | `particle` | none: required where the word has no `given` reading | `false` is required where the word has no `given` reading in any file, and that absence, not the property, is what puts the word in `NON_GIVEN_NAME_PARTICLES`: a particle opening the name starts the family name ([rules/P1](../../docs/design/rules/P1-lone-particle-fold.md), the lone particle fold). `true` is refused, naming the `given` reading to write instead (check 11) | | `may_be_initial` | behavioral | `conjunction` | false | `CONJUNCTIONS_AMBIGUOUS`, on a cased single letter only: read as an initial in a name written in one case, as in `JOHN E SMITH` ([rules/P8](../../docs/design/rules/P8-single-letter-connective-against-the-initial-it-spells.md), a single-letter connective against the initial it spells) | -| `default` | behavioral | `title`, `suffix`, `particle`, `conjunction`, `bound_given`, `relation`, `name` | true | `false` keeps the reading out of every compiled set, `SUFFIX_WORDS` and `GENERATION_WORDS` included for a generation: recorded and checked as a reading, for a pack to turn on in phase 3, where its writing offers the default no cue (`i`, the German regnal ordinal, written `I.`, exactly as an initial is). An exclusion does not refuse it, and it cannot read two ways beside the word's live readings (checks 4 to 6); a name reading beside it marks nothing. On a `given` or `family` reading (3.0's phase 3a) it derives no marker in the default, so the word's other readings compile as they would without it, until a pack turns the name on. It does not count as the name reading an agnomen on by default needs (check 11), and a particle whose word's only `given` readings are off is refused, which side of check 11 it takes being left to the first pack that needs one; a bound given name that is also a particle is refused with it, before check 9 would ask for its `given` reading. `true` is refused: it is what every reading is unwritten | +| `default` | behavioral | `title`, `suffix`, `particle`, `conjunction`, `bound_given`, `relation`, `name` | true | `false` keeps the reading out of every compiled set, `SUFFIX_WORDS` and `GENERATION_WORDS` included for a generation: recorded and checked as a reading, for a choice of languages to turn on (3.0's phase 3), where its writing offers the default no cue (`i`, the German regnal ordinal, written `I.`, exactly as an initial is). An exclusion does not refuse it, and it cannot read two ways beside the word's live readings (checks 4 to 6); a name reading beside it marks nothing. On a `given` or `family` reading (3.0's phase 3a) it derives no marker in the default, so the word's other readings compile as they would without it, until a pack turns the name on. It does not count as the name reading an agnomen on by default needs (check 11), and a particle whose word's only `given` readings are off is refused, which side of check 11 it takes being left to the first pack that needs one; a bound given name that is also a particle is refused with it, before check 9 would ask for its `given` reading. `true` is refused: it is what every reading is unwritten | | `cased` | behavioral | `title`, `suffix`, `particle`, `conjunction`, `bound_given`, `relation`, `name` | the word as written | `CAPITALIZATION_EXCEPTIONS`: the case `capitalized()` gives the word (`PhD`), a mask over its letters and digits, never a replacement (#459); on a name reading, a name whose capitals case repair cannot work out from its spelling (`MacGregor`, #1014) | | `note` | descriptive | every category, a `name` reading included | | free text: why the reading is as it is | | `issue` | descriptive | every category but `name` | | an issue number, or a list of them | @@ -284,20 +284,20 @@ A rule is written only to turn it on: `name_order = "given-first"`, `two_family_ A rule KIND the table does not list (a married-name ending) is a change to the parser, not to a file: a new `Policy` field (with `ADDED_IN`), a row in `build.py`'s `RULE_KEYS` and in `_locale.py`'s `_RULE_VALUES`, and a mechanisms.md entry of its own for the rule kind (mechanisms.md#LOCALE-PACKS-PURE-DATA). `tests/v2/test_vocab.py` holds the two tables to each other and to `Policy`. -## Names reserved for later phases +## Names reserved for later releases -These names will mean something in a later phase. Writing one now as a key, in a file, an entry or a reading, is a build error naming its phase, so nothing reads as working before it does. +These names are reserved for a meaning phase 4 planned and 3.0 did not build: unbuilt, 3.x. Writing one now as a key, in a file, an entry or a reading, is a build error saying so, so nothing reads as working before it does. -| Name | Phase | +| Name | Unbuilt, reserved for | |---|---| -| `bare_only` | 4 | -| `leading_only` | 4 | -| `trailing_only` | 4 | -| `separators` | 4 | +| `bare_only` | 3.x | +| `leading_only` | 3.x | +| `trailing_only` | 3.x | +| `separators` | 3.x | `prefer` and `strict`, reserved for phase 3 by phase 1, are refused for good: phase 3 made each the caller's (a choice of languages prefers its readings, and `parser_for(..., strict=True)` is strict), so the build says a strength is never data. -Phase 2 gave `default` and `attaches` their meaning (2026-10-25, see [Properties](#properties)). The gate's slots carry position and shape already settles a bare letter ([rules/P7](../../docs/design/rules/P7-one-letter-particle-against-the-initial-it-spells.md), a one-letter particle against the initial it spells), so `leading_only`, `trailing_only` and `bare_only` wait for phase 4's shapes, as `separators` waits for its glued words. +Phase 2 gave `default` and `attaches` their meaning (2026-10-25, see [Properties](#properties)). The gate's slots carry position and shape already settles a bare letter ([rules/P7](../../docs/design/rules/P7-one-letter-particle-against-the-initial-it-spells.md), a one-letter particle against the initial it spells), so `leading_only`, `trailing_only` and `bare_only` wait for the shapes phase 4 planned, as `separators` waits for its glued words, none of them built in 3.0. Two property values are refused for the same reason, with a message saying a later phase: `addresses = "full"` and `"any"`, and `attached` with a `side` other than `"after"`. diff --git a/tests/v2/cases.py b/tests/v2/cases.py index edd341e9a..362ae3953 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -3870,8 +3870,10 @@ def _check_cjk_shape_purity(self) -> None: {"given": "John", "middle": "e", "family": "Smith"}, notes="the mixed-case control at three words. The carve-out " "means the ROLE is the same as the row above; the " - "difference is the tag, and so the initials -- 'J. S.' " - "here against 'J. E. S.' for 'JOHN E SMITH'", + "difference is the tag, which no longer moves the " + "initials: since #461 a connective joining nothing " + "initials, so 'J. e. S.' here against 'J. E. S.' for " + "'JOHN E SMITH'", shape=1), Case("one_case_upper_y_joins_as_a_bare_capital", "JUAN GARCIA Y LOPEZ", {"given": "JUAN", "family": "GARCIA Y LOPEZ"}, @@ -3899,10 +3901,12 @@ def _check_cjk_shape_purity(self) -> None: notes="the boundary between the two exceptions: 'Y' is now " "conjunction-tagged rather than initial-tagged, and the " "three-word carve-out still refuses the join, so the " - "ROLE is unchanged. What moves is initials() -- " - "ParsedName.initials() gives 'J. G.' where 2.3 gave " + "ROLE is unchanged. What moved was initials() -- " + "ParsedName.initials() gave 'J. G.' where 2.3 gave " "'J. Y. G.', a conjunction contributing none " - "(rules/R3) -- and the v1 facade's " + "(rules/R3), until #461 gave a connective joining " + "nothing its initial back, 'J. Y. G.' again -- and the " + "v1 facade's " "HumanName.initials() gave 'J. Y. G.' until #528 made " "it read the same tags on 2026-09-13. That is the " "split the 'john e smith' row records, closed on the " @@ -6865,8 +6869,8 @@ def _check_cjk_shape_purity(self) -> None: {"title": "Prof.", "given": "John", "family": "Doe", "suffix": "MA"}, classification="fix(#531)", - notes="MEASURED, and a SECOND field moves: today this is " - "middle 'Prof. MA'. Once 'MA' leaves `walkable` the " + notes="MEASURED, and a SECOND field moves: before #531 this " + "was middle 'Prof. MA'. Once 'MA' leaves `walkable` the " "H5 trailing-title chain reaches 'Prof.' and takes " "it. That is H5's stated transparency working, and it " "lands this row on the same answer as its neighbour " @@ -8473,7 +8477,7 @@ def _check_cjk_shape_purity(self) -> None: classification="fix(#531)", ambiguities=("particle-or-given",), notes="A FAMILY MOVE, found by measurement and accepted by Derek " - "2026-09-18: today this is middle 'van MA', silent. Once " + "2026-09-18: before #531 this was middle 'van MA', silent. Once " "'MA' becomes a suffix, P6's own walk looks past that " "post-nominal, finds 'van' as a trailing all-particle run " "and attaches it. That is exactly 'Berg, Jan van Jr.'s " @@ -8493,8 +8497,8 @@ def _check_cjk_shape_purity(self) -> None: "reporting in the kind naming the reading it OVERRODE, " "which is why its report is `suffix-or-name` where 'Doe, " "John van MA' gives `particle-or-given`; since #785 it is " - "the only one, the capitals settling 'MA' (rules/S6). Today " - "middle 'MA', family 'vd Doe', one report ('Berg, Jan vd' " + "the only one, the capitals settling 'MA' (rules/S6). Before " + "#531 middle 'MA', family 'vd Doe', one report ('Berg, Jan vd' " "is that reading with nothing in front of the particle)"), # ---- policy rows. Through 2.x every one was _CORE_ONLY, a row the # facade runner could not express --------------------------------- @@ -9063,8 +9067,8 @@ def _check_cjk_shape_purity(self) -> None: "reads as its titles and suffixes. Unmoved by #603, which " "extends that reading to a part a credential opens " "whatever stands behind it ('Eric H. Holder, Jr. Chief " - "Justice', 'justice' being no title word, reads suffix " - "'Jr. Justice', title 'Chief')"), + "Witness', 'witness' being no title word, reads suffix " + "'Jr. Witness', title 'Chief')"), Case("family_comma_lone_suffix_piece", "Andrews, M.D.", {"family": "Andrews", "suffix": "M.D."}, classification="fix(comma-family)", @@ -9772,17 +9776,19 @@ def _check_cjk_shape_purity(self) -> None: classification="fix(#810)", ambiguities=("suffix-or-nickname",), notes="the case is asked once, of the name's own words without " - "either clause, so 'MA' stays the nickname where its twin " - "'JOHN SMITH (mba) (MA)' makes the name mixed " - "(rules/S13's Accepted)"), + "either clause, so 'MA' stays the nickname, as it does " + "beside the unambiguous credential in its twin 'JOHN " + "SMITH (dds) (MA)', suffix 'dds' (rules/S13's Accepted); " + "through 2.8 the twin read suffix 'dds MA' (#792)"), Case("bracket_member_settled_by_another_in_a_lower_case_name", "john smith (M.A) (MA)", {"given": "john", "family": "smith", "suffix": "M.A", "nickname": "MA"}, classification="fix(#810)", ambiguities=("suffix-or-nickname",), - notes="as the row above; the twin 'john smith (MBA) (MA)' " - "reads suffix 'MBA MA'"), + notes="as the row above; the twin 'john smith (DDS) (MA)' " + "reads suffix 'DDS', nickname 'MA', where 2.8 read " + "suffix 'DDS MA' (#792)"), Case("bracket_member_past_the_second_comma", "Smith, John, Jr. (M.A)", {"given": "John", "family": "Smith", "suffix": "Jr. M.A"}, classification="fix(#810)", @@ -9823,7 +9829,8 @@ def _check_cjk_shape_purity(self) -> None: classification="fix(#810)", ambiguities=("suffix-or-nickname",), notes="no name word, so no clause after the name; the twin " - "'Dr. (MBA)' reads given 'Dr.'"), + "'Dr. (DDS)' reads title 'Dr.', suffix 'DDS', where 2.8 " + "read given 'Dr.' (#792)"), Case("bracket_member_in_quotes", 'John Smith "MA"', {"given": "John", "family": "Smith", "suffix": "MA"}, notes="every default nickname pair escapes alike (rules/S1)"), @@ -9831,8 +9838,9 @@ def _check_cjk_shape_purity(self) -> None: {"family": "Jack", "suffix": "MA"}, classification="fix(#810)", notes="the word stays N3's family name, read beside the clause " - "as a nickname, where its twin 'Jack (PhD)' reads given " - "'Jack' (rules/S13's Accepted); 1.4.0 read last 'MA', " + "as a nickname, as its twin 'Jack (PhD)' reads family " + "'Jack' (rules/S13's Accepted), where 2.8 read given " + "'Jack' (#792); 1.4.0 read last 'MA', " "2.0.0 through 2.6.0 family 'Jack', nickname 'MA'"), Case("bracket_member_with_a_period_after_a_lone_name_word", "Jack (M.A)", @@ -15263,8 +15271,9 @@ def _check_cjk_shape_purity(self) -> None: "nickname": "マイケル ジャクソン"}, classification="fix(#272)", notes="delimited content tokenizes under the same separator " - "rules: the dot renders back as a space in the nickname " - "join -- a decision, not an accident"), + "rules: the dot reads back as a space in the nickname " + "field's join, str() keeping it as written (rules/R6) " + "-- a decision, not an accident"), Case("zh_interpunct_transcription_source_order", "威廉·莎士比亚", {"given": "威廉", "family": "莎士比亚"}, classification="fix(#298)", diff --git a/tests/v2/rules_budget.json b/tests/v2/rules_budget.json index d67202b63..1dbb3d080 100644 --- a/tests/v2/rules_budget.json +++ b/tests/v2/rules_budget.json @@ -26,7 +26,7 @@ "H6": {"words": 323, "then_words": 79, "because_sentences": 3, "except": 4, "precedence": 0, "accepted": 2, "accepted_words": 114}, "H7": {"words": 339, "then_words": 270, "because_sentences": 2, "except": 0, "precedence": 0, "accepted": 1, "accepted_words": 54}, "H8": {"words": 431, "then_words": 68, "because_sentences": 3, "except": 3, "precedence": 3, "accepted": 3, "accepted_words": 131}, - "H9": {"words": 778, "then_words": 292, "because_sentences": 3, "except": 0, "precedence": 6, "accepted": 11, "accepted_words": 619}, + "H9": {"words": 784, "then_words": 292, "because_sentences": 3, "except": 0, "precedence": 6, "accepted": 11, "accepted_words": 619}, "M1": {"words": 239, "then_words": 119, "because_sentences": 2, "except": 1, "precedence": 2, "accepted": 0, "accepted_words": 0}, "M2": {"words": 755, "then_words": 435, "because_sentences": 1, "except": 4, "precedence": 1, "accepted": 6, "accepted_words": 204}, "M3": {"words": 239, "then_words": 84, "because_sentences": 2, "except": 2, "precedence": 2, "accepted": 1, "accepted_words": 93}, diff --git a/tests/v2/test_lexicon.py b/tests/v2/test_lexicon.py index 3ad002e02..9517f14dd 100644 --- a/tests/v2/test_lexicon.py +++ b/tests/v2/test_lexicon.py @@ -993,6 +993,59 @@ def test_a_2x_spelling_is_refused_with_calls_that_run( assert isinstance(built, Lexicon) +@pytest.mark.parametrize("call", [ + lambda: Lexicon(surnames={"kim"}, titles={"dr"}), # type: ignore[call-arg] + lambda: Lexicon(given_names={"luis"}, # type: ignore[call-arg] + capitalization_exceptions={"phd": "PhD"}, titles={"dr"}), + lambda: Lexicon(titles={"dr"}), # type: ignore[call-arg] +]) +def test_a_2x_constructor_spelling_offers_calls_on_a_lexicon_it_names( + call: Callable[[], object]) -> None: + # a bare Lexicon(...) call binds no variable, so the offered code + # must build the lexicon it was asked for with no `lexicon` in scope + with pytest.raises(TypeError, match="since 3.0") as raised: + call() + code = str(raised.value).rsplit("write ", 1)[1] + assert not code.startswith("lexicon"), code + built = eval(code, {"Lexicon": Lexicon}) # noqa: S307 + assert isinstance(built, Lexicon) + assert "dr" in built._sets.titles + # the lists and masks the call passed are kept + if "kim" in code: + assert built.surnames == frozenset({"kim"}) + if "luis" in code: + assert built.given_names == frozenset({"luis"}) + assert built.capitalization_exceptions == (("phd", "PhD"),) + + +def test_a_2x_replace_of_an_edited_lexicon_offers_calls_on_that_lexicon( + ) -> None: + # dataclasses.replace shares Lexicon()'s refusal; an edited lexicon + # is the caller's variable, which no constructor call rebuilds + edited = Lexicon.empty().add("zzx", kind="title") + with pytest.raises(TypeError, match="since 3.0") as raised: + dataclasses.replace(edited, titles={"zzq"}) # type: ignore[call-arg] + code = str(raised.value).rsplit("write ", 1)[1] + assert code.startswith("lexicon."), code + built = eval(code, {"lexicon": edited}) # noqa: S307 + assert built._sets.titles == frozenset({"zzq"}) + + +def test_a_2x_removal_keeps_another_files_reading_of_the_word() -> None: + # ma is a credential (en) and a Filipino given name written short for + # maría (fil): taking out the credential must not rewrite the + # Filipino reading's full name + with pytest.raises(TypeError, match="since 3.0") as raised: + Lexicon.default().remove(suffix_acronyms={"ma"}) # type: ignore[call-arg] + code = str(raised.value).rsplit("write ", 1)[1] + assert code == "lexicon.remove('ma', kind='credential')", code + built = eval(code, {"lexicon": Lexicon.default()}) # noqa: S307 + assert "ma" not in built._sets.suffix_acronyms + assert "ma" in built._sets.given_name_abbreviations + assert any(r.kind == "given" and dict(r.properties).get("same_as") + == "maría" for r in built.readings_of("ma")) + + def test_a_2x_spelling_with_no_exact_reading_offers_the_nearest_that_runs( ) -> None: # `phd` both an acronym and a word in 2.x: a word has one suffix diff --git a/tests/v2/test_vocab.py b/tests/v2/test_vocab.py index 818f8b318..45976ed5a 100644 --- a/tests/v2/test_vocab.py +++ b/tests/v2/test_vocab.py @@ -255,7 +255,7 @@ def _w(body: str) -> str: ("no-schema", {"und.toml": '[[word]]\ntext = "jr"\ngeneration = {}\n'}, "needs `schema = 1`"), ("newer-schema", {"und.toml": OK.replace("schema = 1", "schema = 2")}, "newer than this compiler"), ("unknown-top-level-key", {"und.toml": "language = 'en'\n" + OK}, "unknown key 'language'"), - ("reserved-top-level-key", {"und.toml": "separators = []\n" + OK}, "reserved for phase 4"), + ("reserved-top-level-key", {"und.toml": "separators = []\n" + OK}, "reserved, unbuilt, for 3.x"), # phase 3 gave the strengths to the caller, so they stay refused as # data, the message naming the call (3j) ("prefer-is-not-data", {"und.toml": "prefer = ['pt']\n" + OK}, "'prefer' is not data; a choice"), @@ -263,7 +263,7 @@ def _w(body: str) -> str: ("file-name-not-a-tag", {"english.toml": OK}, "BCP 47 tag"), ("file-name-not-canonical-case", {"pt-br.toml": OK}, "BCP 47 tag"), ("file-with-no-entries", {"en.toml": OK, "und.toml": "schema = 1\n"}, "no \\[\\[word\\]\\] entries"), - ("reserved-reading-property", {"und.toml": _w('[[word]]\ntext = "mba"\ncredential = { bare_only = true }')}, "reserved for phase 4"), + ("reserved-reading-property", {"und.toml": _w('[[word]]\ntext = "mba"\ncredential = { bare_only = true }')}, "reserved, unbuilt, for 3.x"), ("unknown-reading-property", {"und.toml": _w('[[word]]\ntext = "mba"\ncredential = { acronymm = true }')}, "unknown key 'acronymm'"), ("unknown-word-key", {"und.toml": _w('[[word]]\ntext = "jr"\ngeneration = {}\nlang = "en"')}, "unknown key 'lang'"), ("descriptive-key-on-the-word", {"und.toml": _w('[[word]]\ntext = "jr"\ngeneration = {}\nnote = "x"')}, "belongs on a reading"), @@ -341,7 +341,8 @@ def _w(body: str) -> str: ("flag-not-a-bool", {"und.toml": _w('[[word]]\ntext = "van"\nparticle = { may_be_given = "yes" }')}, "true or false"), ("issue-not-a-number", {"en.toml": _w('[[word]]\ntext = "jr"\ngeneration = { issue = "661" }')}, "issue number"), # `default`: a reading is on unwritten, so only false is worth - # writing; and the four names phase 2 left reserved say which phase + # writing; and the four names phase 2 left reserved say they are + # unbuilt, for 3.x ("default-true", {"en.toml": _w('[[word]]\ntext = "jr"\ngeneration = { default = true }')}, "write default = false"), ("default-not-a-bool", {"en.toml": _w('[[word]]\ntext = "jr"\ngeneration = { default = "no" }')}, "write default = false"), ("completed_by-not-a-list", {"ar.toml": _w('[[word]]\ntext = "zzq"\nbound_given = { completed_by = "zzr" }')}, "completed_by must be a non-empty list"), @@ -357,7 +358,7 @@ def _w(body: str) -> str: ("attaches-a-word-with-no-title-reading", {"en.toml": _w('[[word]]\ntext = "chief"\noffice = { attaches = ["zzq"] }')}, "attaches names 'zzq', which is no title kind and no title word"), ("attaches-on-a-suffix", {"en.toml": _w('[[word]]\ntext = "phd"\ncredential = { attaches = ["office"] }')}, "unknown key 'attaches'"), ("attaches-differs-between-files", {"en.toml": _w('[[word]]\ntext = "chief"\noffice = { attaches = ["military"] }'), "fr.toml": _w('[[word]]\ntext = "chief"\noffice = {}')}, "attaches"), - ("reserved-separators", {"th.toml": _w('[[word]]\ntext = "นาย"\nsalutation = { separators = [] }')}, "reserved for phase 4"), + ("reserved-separators", {"th.toml": _w('[[word]]\ntext = "นาย"\nsalutation = { separators = [] }')}, "reserved, unbuilt, for 3.x"), # 11, the agnomen's half: an agnomen is gated by its name reading, # so one without would take the family name from two-word names ("name-reading-beside-an-honorific", {"ja.toml": _w('[[word]]\ntext = "zzt"\nhonorific = {}\nfamily = { note = "x" }')}, "gates nothing"), @@ -1228,9 +1229,9 @@ def test_the_readme_lists_the_name_reading_keys() -> None: assert len(rows) == len(build.NAME_KEYS) -def test_the_readme_lists_the_reserved_names_with_their_phases() -> None: - rows = _readme_table("## Names reserved for later phases") - assert {_code(r[0])[0]: int(r[1]) for r in rows} == build.RESERVED +def test_the_readme_lists_the_reserved_names_with_their_releases() -> None: + rows = _readme_table("## Names reserved for later releases") + assert {_code(r[0])[0]: r[1] for r in rows} == build.RESERVED assert len(rows) == len(build.RESERVED) diff --git a/tools/differential/corpus_rules.jsonl b/tools/differential/corpus_rules.jsonl index a89401766..64936b71d 100644 --- a/tools/differential/corpus_rules.jsonl +++ b/tools/differential/corpus_rules.jsonl @@ -1018,6 +1018,7 @@ "SEÁN Ó MURCHÚ" "SIG.RA MARIA ROSSI" "SMITH VD MA, JOHN" +"SMITH VD V, JOHN" "SMITH VD, JOHN" "SMITH, DEAN MR." "SMITH, JOHN B. MAJ." @@ -1350,6 +1351,7 @@ "née Jones" "oglu Ali" "qc mp" +"smith vd v, john" "smith, de ma" "smith, j y lopez" "smith, john e phd" diff --git a/tools/differential/snapshot.jsonl b/tools/differential/snapshot.jsonl index c8ac8fb27..b7f9e8527 100644 --- a/tools/differential/snapshot.jsonl +++ b/tools/differential/snapshot.jsonl @@ -3322,6 +3322,7 @@ {"name": "SEÁN Ó MURCHÚ", "given": "SEÁN", "family": "Ó MURCHÚ", "initials": "S. M.", "capitalized": "Seán Ó Murchú"} {"name": "SIG.RA MARIA ROSSI", "title": "SIG.RA", "given": "MARIA", "family": "ROSSI", "initials": "M. R.", "capitalized": "Sig.ra Maria Rossi"} {"name": "SMITH VD MA, JOHN", "given": "JOHN", "family": "SMITH VD MA", "initials": "J. S. M.", "capitalized": "John Smith vd Ma"} +{"name": "SMITH VD V, JOHN", "given": "JOHN", "family": "SMITH VD", "suffix": "V", "ambiguities": ["SUFFIX_OR_NAME"], "alternatives": ["SUFFIX_OR_NAME: initial as family (shape)"], "initials": "J. S.", "capitalized": "John Smith vd V", "generation": "V"} {"name": "SMITH VD, JOHN", "given": "JOHN", "family": "SMITH", "suffix": "VD", "initials": "J. S.", "capitalized": "John Smith VD"} {"name": "SMITH, DEAN MR.", "title": "MR.", "given": "DEAN", "family": "SMITH", "initials": "D. S.", "capitalized": "Mr. Dean Smith"} {"name": "SMITH, JOHN B. MAJ.", "title": "MAJ.", "given": "JOHN", "middle": "B.", "family": "SMITH", "initials": "J. B. S.", "capitalized": "Maj. John B. Smith"} @@ -4355,6 +4356,7 @@ {"name": "scott e. werner", "given": "scott", "middle": "e.", "family": "werner", "initials": "s. e. w.", "capitalized": "Scott E. Werner"} {"name": "scott l. werner", "given": "scott", "middle": "l.", "family": "werner", "initials": "s. l. w.", "capitalized": "Scott L. Werner"} {"name": "señora María García", "title": "señora", "given": "María", "family": "García", "initials": "M. G.", "capitalized_force": "Señora María García"} +{"name": "smith vd v, john", "given": "john", "family": "smith vd", "suffix": "v", "ambiguities": ["SUFFIX_OR_NAME"], "alternatives": ["SUFFIX_OR_NAME: initial as family (shape)"], "initials": "j. s.", "capitalized": "John Smith vd V", "generation": "v"} {"name": "smith, de ma", "family": "smith de ma", "ambiguities": ["SUFFIX_OR_NAME"], "alternatives": ["SUFFIX_OR_NAME: credential as suffix (vocabulary)"], "initials": "s. m.", "capitalized": "Smith de Ma"} {"name": "smith, j y lopez", "given": "j", "middle": "y lopez", "family": "smith", "initials": "j. l. s.", "capitalized": "J y Lopez Smith"} {"name": "smith, john e phd", "given": "john", "middle": "e", "family": "smith", "suffix": "phd", "initials": "j. e. s.", "capitalized": "John E Smith PhD"} diff --git a/tools/vocab/build.py b/tools/vocab/build.py index 62627728e..8d3122152 100644 --- a/tools/vocab/build.py +++ b/tools/vocab/build.py @@ -272,14 +272,16 @@ def _alnum(text: str) -> str: #: The keys of an `excluded` value's table, and of an `attached` table. EXCLUSION_KEYS = frozenset({"reason", "issue", "decided", "note"}) ATTACHED_KEYS = frozenset({"side", "issue", "decided", "note"}) -#: Names a later phase will give meaning to. Writing one now is an error -#: that says so, rather than "unknown", so nothing reads as working. +#: Names reserved for a meaning not yet built. Writing one now is an +#: error that says so, rather than "unknown", so nothing reads as working. #: Phase 2 gave `default` and `attaches` their meaning; the gate's slots #: carry position, and shape settles a bare letter (rules/P7), so the -#: position and bare-letter names wait with `separators` for phase 4's -#: shapes and glued words. +#: position and bare-letter names wait with `separators` for the shapes +#: and glued words phase 4 planned, unbuilt in 3.0 and left to a 3.x +#: release. RESERVED = { - "bare_only": 4, "leading_only": 4, "trailing_only": 4, "separators": 4, + "bare_only": "3.x", "leading_only": "3.x", "trailing_only": "3.x", + "separators": "3.x", } #: Names phase 1 reserved for phase 3 that phase 3 gave to the caller #: instead: a choice prefers (3b) and `parser_for(..., strict=True)` is @@ -410,8 +412,8 @@ class Pool: # ---------------------------------------------------------------- loading def _reserved_or_unknown(where: str, key: str, allowed: Iterable[str]) -> VocabError: if key in RESERVED: - return VocabError(f"{where}: {key!r} is reserved for phase {RESERVED[key]}; " - f"it has no behavior yet") + return VocabError(f"{where}: {key!r} is reserved, unbuilt, for " + f"{RESERVED[key]}; it has no behavior yet") if key in NOT_DATA: return VocabError(f"{where}: {key!r} is not data; {NOT_DATA[key]}") return VocabError(f"{where}: unknown key {key!r}; allowed: {sorted(allowed)}")