diff --git a/AGENTS.md b/AGENTS.md index bf59f59ea..291352bb6 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -32,7 +32,7 @@ Three committed contributor docs carry the parser's normative rules and their re **Counting claims.** A bare count in prose is either an assertion or a liability, keyed by who observes its staleness: asserted counts (a test holds the number) fail CI at change time — the useful kind; dated snapshots ("51 sites at spec time") cannot go stale; standing present-tense prose counts are the forbidden class — promote to an assertion, add a date, or state the invariant and let a test count. After changing how many times something runs, sweep for counts, not for the thing's name. -**Release-log claims.** Quantified or universal behavior claims in release bullets must come from the snapshot's diff since the last release, be verified against rules examples, or -- for a view the snapshot cannot see -- carry a recompute recipe stored with the design entry the bullet cites; never write one from memory. `uv run python tools/differential/snapshot.py --since v` prints every corpus name the release moved, grouped by changed field; v2.4.1 is the first tag carrying a snapshot, and for an older release `compare.py --baseline ` prints the same report against the PyPI wheel, closing with the entries compared and how many moved names are Latin-only. Both say how many moved names only `corpus_cldr.jsonl` holds. The snapshot records the seven role fields, ambiguity kinds, `initials()`, the two case-repair views (`capitalized()` and `capitalized(force=True)`, each only where it changes the text, #826) and, since 3.0, the suffix's `generation` and `credentials` views (where the suffix holds a generation, rules/R8), its `honorifics` view (where it is not empty, rules/R8, #326) and the `patronymic` and `surname2` views (each where it is not empty, rules/R9 and R10; the snapshot parses the default policy, so no name fills `surname2`, which `two_family_names` reads and, under `locales.ZH`, a Han surname pair, rules/W5) for every name, so an initials, case, generation or patronymic count is read off it directly -- with two limits. A case view holds the whole repaired name, so in a name repair changes, a role move that changes the rendering moves the view too, with or without a word recased: read a view move that comes with a role move by its words' case, not by whether it moved (#808's `John Smith, otr/l` moved both, `'John Smith OTR/L'` -> `'Otr/L John Smith'`). And `--since` a tag whose snapshot predates #826 (v2.5.0 and earlier) does not compare the views at all, so 2.6.0's case claims come from `compare.py --baseline 2.5.0`. `corpus_cldr.jsonl`'s fictional renderings are in it: count them out of a release figure (both tools print how many moved names only that file holds), or say the figure includes them. A render view the snapshot does not record -- `str()` under a custom format, `initials()` under a custom spec -- stays invisible to it, and for those the first two sources cannot reach a claim: the snapshot is byte-identical across the change, and an example line witnesses an output without counting anything. The DEFAULT `str()` is mostly invisible too, the snapshot seeing it only through a case view where repair changes the text: #754's render defaults moved `str()` for 1,034 corpus entries and the case views for 159, so a change to the default rendering is measured by rendering every entry in both trees (decisions/R6's 2026-10-29 entry holds the recipe). A recipe names the corpus files, the policy sweep, and -- the part that is easy to omit and fatal -- THE COMPARATOR, which must be something the shipped tree is not: #408's first recipe said to compare `initials()` against a folded-first partition, which is what `initials()` now IS, so it reproduced 0 where the bullet claimed 660 and was the only stated provenance for the number. Run the recipe as written before shipping the bullet. Cross-version numbers (a released wheel, the pre-change tree) are dated snapshots under Counting claims, since nothing in the repository re-runs them. The same standard binds decisions entries and PR bodies: a claim that a change restores the released reading is measured over the whole moved population against the wheel, never spot-checked (#625 claimed its 774 moves landed on the 1.4.0 and 2.3.0 readings from three names; measured, the tree matched 2.3.0 on 0 of 258 moved texts, master on 152), and a population size is re-counted from its recipe, never copied from another entry (the same entry carried #624's 137,810 for a grid of 395,148). Code comments citing rule IDs follow the excerpt discipline, checked by tests/v2/test_doc_citations.py: a rule's quotable text is its fields' prose (When, Tolerated, Then and its bullets, Because, and the Except, Precedence and Accepted bullets), and its fences and the footer below `---`, where the `Signal:` line stands, are not; `rules/H` names the H section, whose Background is quotable; a decisions citation may paraphrase, but what it quotes must be verbatim; and a reference with no quote must still name something that exists. The check's reach is a quote within four words of the ID or chained to one that is: a quote further off reads exactly like a name written in double quotes, so it is NOT checked, and the second review of #632 found 14 such quotes stale by hand. Keep a quote beside its ID. Quotes of AGENTS.md itself are not checked at all. **A citation comment holds three things: the quote, a line saying which clause this code implements, and `decisions/X (#N)` pointers to the entries that hold the history.** A code-local reason a reader of that line needs may stay beside them (why an index is safe to keep, why a test runs last), but issue history, measurements and restated examples belong in decisions/, where they are dated and kept current; in a comment they are a second copy that goes stale where nobody looks. #760 trimmed five blocks of 59 to 100 lines that had grown that way (`_post_rules._mark_suffix_entries`, group's P3 both-sides test, its S2/P2/P4 chain head and S4 chain emitter, and the honorific peel's scope in `_script_segment`), moving into dated decisions bullets what those entries lacked. Two of the moved claims no longer held: a policy example that had stopped reaching its case, and a credential said to land as a title that lands as a suffix. +**Release-log claims.** Quantified or universal behavior claims in release bullets must come from the snapshot's diff since the last release, be verified against rules examples, or -- for a view the snapshot cannot see -- carry a recompute recipe stored with the design entry the bullet cites; never write one from memory. `uv run python tools/differential/snapshot.py --since v` prints every corpus name the release moved, grouped by changed field; v2.4.1 is the first tag carrying a snapshot, and for an older release `compare.py --baseline ` prints the same report against the PyPI wheel, closing with the entries compared and how many moved names are Latin-only. Both say how many moved names only `corpus_cldr.jsonl` holds. The snapshot records the seven role fields, ambiguity kinds, each report's alternative reading (since #752's phase 2m), `initials()`, the two case-repair views (`capitalized()` and `capitalized(force=True)`, each only where it changes the text, #826) and, since 3.0, the suffix's `generation` and `credentials` views (where the suffix holds a generation, rules/R8), its `honorifics` view (where it is not empty, rules/R8, #326) and the `patronymic` and `surname2` views (each where it is not empty, rules/R9 and R10; the snapshot parses the default policy, so no name fills `surname2`, which `two_family_names` reads and, under `locales.ZH`, a Han surname pair, rules/W5) for every name, so an initials, case, generation or patronymic count is read off it directly -- with two limits. A case view holds the whole repaired name, so in a name repair changes, a role move that changes the rendering moves the view too, with or without a word recased: read a view move that comes with a role move by its words' case, not by whether it moved (#808's `John Smith, otr/l` moved both, `'John Smith OTR/L'` -> `'Otr/L John Smith'`). And `--since` a tag whose snapshot predates #826 (v2.5.0 and earlier) does not compare the views at all, so 2.6.0's case claims come from `compare.py --baseline 2.5.0`. `corpus_cldr.jsonl`'s fictional renderings are in it: count them out of a release figure (both tools print how many moved names only that file holds), or say the figure includes them. A render view the snapshot does not record -- `str()` under a custom format, `initials()` under a custom spec -- stays invisible to it, and for those the first two sources cannot reach a claim: the snapshot is byte-identical across the change, and an example line witnesses an output without counting anything. The DEFAULT `str()` is mostly invisible too, the snapshot seeing it only through a case view where repair changes the text: #754's render defaults moved `str()` for 1,034 corpus entries and the case views for 159, so a change to the default rendering is measured by rendering every entry in both trees (decisions/R6's 2026-10-29 entry holds the recipe). A recipe names the corpus files, the policy sweep, and -- the part that is easy to omit and fatal -- THE COMPARATOR, which must be something the shipped tree is not: #408's first recipe said to compare `initials()` against a folded-first partition, which is what `initials()` now IS, so it reproduced 0 where the bullet claimed 660 and was the only stated provenance for the number. Run the recipe as written before shipping the bullet. Cross-version numbers (a released wheel, the pre-change tree) are dated snapshots under Counting claims, since nothing in the repository re-runs them. The same standard binds decisions entries and PR bodies: a claim that a change restores the released reading is measured over the whole moved population against the wheel, never spot-checked (#625 claimed its 774 moves landed on the 1.4.0 and 2.3.0 readings from three names; measured, the tree matched 2.3.0 on 0 of 258 moved texts, master on 152), and a population size is re-counted from its recipe, never copied from another entry (the same entry carried #624's 137,810 for a grid of 395,148). Code comments citing rule IDs follow the excerpt discipline, checked by tests/v2/test_doc_citations.py: a rule's quotable text is its fields' prose (When, Tolerated, Then and its bullets, Because, and the Except, Precedence and Accepted bullets), and its fences and the footer below `---`, where the `Signal:` line stands, are not; `rules/H` names the H section, whose Background is quotable; a decisions citation may paraphrase, but what it quotes must be verbatim; and a reference with no quote must still name something that exists. The check's reach is a quote within four words of the ID or chained to one that is: a quote further off reads exactly like a name written in double quotes, so it is NOT checked, and the second review of #632 found 14 such quotes stale by hand. Keep a quote beside its ID. Quotes of AGENTS.md itself are not checked at all. **A citation comment holds three things: the quote, a line saying which clause this code implements, and `decisions/X (#N)` pointers to the entries that hold the history.** A code-local reason a reader of that line needs may stay beside them (why an index is safe to keep, why a test runs last), but issue history, measurements and restated examples belong in decisions/, where they are dated and kept current; in a comment they are a second copy that goes stale where nobody looks. #760 trimmed five blocks of 59 to 100 lines that had grown that way (`_post_rules._mark_suffix_entries`, group's P3 both-sides test, its S2/P2/P4 chain head and S4 chain emitter, and the honorific peel's scope in `_script_segment`), moving into dated decisions bullets what those entries lacked. Two of the moved claims no longer held: a policy example that had stopped reaching its case, and a credential said to land as a title that lands as a suffix. **Writing the user docs (docs/*.rst) has its own AGENTS.md too.** `docs/AGENTS.md` carries the style the docs are written in — tables as indexes, subheadings per task, measured claims, recipe doctests — distilled from PR #588. It loads the same way as the docs/design/ one below; if your tool does not do nested discovery, read it before editing a `.rst` file under docs/. @@ -73,7 +73,9 @@ uv run ruff check # Debug how a specific name string is parsed: 2.0 core parse() -- # prints the ParsedName repr raw and capitalized, then initials; -# --json emits the component dict +# --json emits the component dict; --locale/--strict/--order, +# --lone-name-word and the Policy switch flags parse under a +# configuration (--help lists them) uv run python -m nameparser "Dr. Juan Q. Xavier de la Vega III" # Rebuild the compiled vocabulary after editing nameparser/vocab/*.toml @@ -144,9 +146,10 @@ uv run sphinx-build -b html docs dist/docs # on the module (a vocabulary set is read as `lexicon._sets.titles` # since 4i-a, and a Reads: line names it `Lexicon.titles`). A Reads: # line may name a set of fields at once: _classify's "every Lexicon -# vocabulary field except surnames and honorific_tails" stands for -# twenty, 17 read in the module (the grep lists them) and the rest -# through _vocab's helpers, so check what it leaves out +# vocabulary field except surnames and honorific_tails, ... and +# given_names" stands for every `_lexicon._VOCAB_FIELDS` entry but +# those three, most read in the module (the grep lists them) and the +# rest through _vocab's helpers, so check what it leaves out # - tests/v2/cases.py notes, which explain why a row lands where it does # - docs/design/rules/, decisions/ and mechanisms.md -- READ, don't # grep: the excerpt and example tests catch citation and example drift, @@ -267,7 +270,7 @@ The 2.0 rewrite landed as underscore-private modules beside the v1 code, which 2 - **A reading is of one word, save a birth marker and a title addressing the given name** (the build's check 8, which `_lexicon._make_edit` runs on every `add` and RAISES on, since a reading is now a caller's statement the compile can refuse as the build does: 3.0's 4i-b). The two lists, `surnames` and `given_names`, and `capitalization_exceptions` keys still WARN on a multi-word entry (`_normset`/`_normpairs` via `_warn_at_caller`, UserWarning, never a raise — an inert entry produces nothing). The default vocabulary and every locale pack must stay warning-free (`test_default_lexicon_builds_warning_free`, `test_pack_vocabulary_entries_are_single_words`). - **Invariants guard harm, not no-ops**: add a constructor check when violating it produces a *wrong parse*, not when it produces *nothing*. A false positive costs a working configuration; a true positive on an inert condition costs the user nothing, so that trade is never worth taking. Since 3.0's 4i-b a `Lexicon` compiles from readings, which hold the 2.x marker-over-base relations by construction; the checks a caller's edit reaches are the build's checks 3 (a kind and its properties, `may_be_initial` on a single cased letter alone), 4 (one suffix kind per word), 8 (a phrase), 9 (a bound given name that is a particle has a `given` reading) and 11's agnomen clause (an agnomen has a name reading), in `_lexicon._make_edit` and `_check_pool`, which `_lexicon._built` runs again under a choice of languages, refusing a caller's reading that clashes with one the choice turns on and naming the edit, each raising with the call that fixes it, because each is a wrong parse. A given-addressing title run the parse can never build is not checked, because an unreachable entry is simply never consulted (see Gotchas). Before adding one, construct the config it forbids and check what actually breaks. **Between raise and silence sits the construction-time `UserWarning`**, for a gap that is statically decidable, harmless to SOME deliberate caller, and indistinguishable-from-working for everyone else: the segmenterless activation (#337 — `parser_for(locales.JA)` without a segmenter behaved exactly like a working parser minus the feature) warns rather than raises because the inert JA registration is itself a pinned property, and a warning is filterable by the caller who wants exactly that. The message must carry every applicable remedy and no inapplicable one (the `ja_segmenter` hint fires only when a Japanese script is among the dead ones). Test fuzzers that legitimately construct such configs suppress the warning by MESSAGE, never by category — a blanket `UserWarning` ignore would mask the next construction diagnostic (`_quiet_parser` in `tests/v2/test_properties.py` is the pattern). - **Reprs are bounded**: render which fields deviate from a named baseline and by how much, never contents (`Lexicon(default + titles: +2)`). `PolicyPatch`'s repr shows only set (non-UNSET) fields; `_order_repr` must never raise even on an unvalidated patch's garbage `name_order` (PolicyPatch defers validation to apply time); the sweep test in `tests/v2/test_reprs.py` pins that no config repr leaks the UNSET sentinel. -- **Every pipeline stage is one module holding one public function of the same name** (`_tokenize.py`/`tokenize`), with the exceptions named where they stand — `_group.py`'s `marker_run_length`, a shared predicate (mechanisms.md#ONE-PREDICATE-PER-QUESTION), and since #511 `_post_rules.py`'s `suffix_entries`, the R1 entry pass as a function, because `Parser.revise` runs that one pass over a forced-role sub-parse and a stage's tail cannot be called on its own — **and its module docstring declares the contract in three labelled lines**: `Consumes:` what it takes from `ParseState`, `Produces:` what it hands back, `Reads:` which `Policy`/`Lexicon` fields it consults. `Reads:` is the load-bearing one — it makes "which stage do I touch for this feature?" a grep rather than a read-through. A `Reads:` line names a vocabulary set as `Lexicon.`, and every library module but `_lexicon` reads it as `lexicon._sets.` (`_lexicon._LexiconSets`, #754's 4i-a), never off the public field, `surnames` and `given_names` included though they stay public, which let 4i-b make the other twenty internal without touching a reader; `tests/v2/test_layering.py` refuses a direct read, field by field, in the library and in every tool. The authority for the stage set is `_pipeline/__init__.py`'s `STAGES` (eight, and not public API) with the field-ownership map in `ParseState`'s docstring, pinned by `tests/v2/pipeline/test_state.py`; NOT the rules, which are implementation-free by their own preamble and whose `implemented:` names modules honoring a rule rather than stages. A `_pipeline/` module that is not a stage says so in its first line instead (`_assemble.py`: "Not a stage: …", with a `Reads:` line all the same, since it names a reported field by `Policy.middle_as_family` and a token's reading by three `Lexicon` sets), so absence of the three lines is a claim about the module rather than an oversight. Provenance: §5 of the 2026-07-11 conventions spec, recorded here 2026-08-16. +- **Every pipeline stage is one module holding one public function of the same name** (`_tokenize.py`/`tokenize`), with the exceptions named where they stand — `_group.py`'s `marker_run_length`, a shared predicate (mechanisms.md#ONE-PREDICATE-PER-QUESTION), and since #511 `_post_rules.py`'s `suffix_entries`, the R1 entry pass as a function, because `Parser.revise` runs that one pass over a forced-role sub-parse and a stage's tail cannot be called on its own — **and its module docstring declares the contract in three labelled lines**: `Consumes:` what it takes from `ParseState`, `Produces:` what it hands back, `Reads:` which `Policy`/`Lexicon` fields it consults. `Reads:` is the load-bearing one — it makes "which stage do I touch for this feature?" a grep rather than a read-through. A `Reads:` line names a vocabulary set as `Lexicon.`, and every library module but `_lexicon` reads it as `lexicon._sets.` (`_lexicon._LexiconSets`, #754's 4i-a), never off the public field, `surnames` and `given_names` included though they stay public, which let 4i-b make the other twenty internal without touching a reader; `tests/v2/test_layering.py` refuses a direct read, field by field, in the library and in every tool. The authority for the stage set is `_pipeline/__init__.py`'s `STAGES` (eight, and not public API) with the field-ownership map in `ParseState`'s docstring, pinned by `tests/v2/pipeline/test_state.py`; NOT the rules, which are implementation-free by their own preamble and whose `implemented:` names modules honoring a rule rather than stages. A `_pipeline/` module that is not a stage says so in its first line instead (`_assemble.py`: "Not a stage: …", with a `Reads:` line all the same, since it reads `Policy` and `Lexicon` fields too (its `Reads:` line lists them)), so absence of the three lines is a claim about the module rather than an oversight. Provenance: §5 of the 2026-07-11 conventions spec, recorded here 2026-08-16. - **Frames are a budget for hot and scaling paths; everywhere else, prefer the shared predicate.** The parser's cost is counted in Python frames (`sys.setprofile` "call" events — deterministic and load-immune, which is why #475 chose them over wall-clock timing), but one frame is small: measured 2026-09-28 on py3.11, a trivial call costs about 13 ns against 40–80 µs for a whole parse, so one frame per parse is roughly 0.03% of it. What the budget exists to catch is how a cost SCALES and how it ACCUMULATES, so weigh an extra frame by four questions, in order. (1) Does it multiply with the input — per token, per member, per pair? Then it matters, and a hand-inlined fast path can earn its place (the inline tag tests ahead of `_pieces.starts_a_credential_run` in its loops, each a necessary half of the predicate so nothing can drift; `_vocab.run_word_fold` was the measured example until #613 deleted the run test it served). Folding a copy into a shared helper is where such a pre-check gets lost: #617 replaced `_comma._title_chains`, whose loop tested the conjunction tag inline, with a call into group's loop, which asks `is_conj_piece` per piece, and every family-comma parse paid two frames per word of the part (`Smith, John` +2) while the reference row fell -- review caught it; the fix gates the call on the tag walk the caller already makes, and `test_a_comma_part_with_no_free_connective_skips_the_join` holds it. Count a comma name, not only the reference row, when a fold touches the comma path. (2) How many names pay it — every parse, where plain names like `Smith, John` must keep their cost, or only a rare shape? (3) Does it move an enforced guard — the ±2% reference band in `test_benchmark.py` (`_CALL_BASELINE`, decisions/parse-cost) or a scaling ratio? (4) What does the inline alternative cost to maintain? A hand copy of a predicate that can drift is the defect class mechanisms.md#ONE-PREDICATE-PER-QUESTION names, and it compounds; a NARROWER question stated exactly is not a copy and may stay inline on that ground alone. A constant frame on a rare shape that no guard sees loses to readability: #544's cleanup declined `is_title_piece` for one frame on comma credential runs (24 ns a parse, no guard moved), a stricter call than this rule asks for; by this rule the inline test is still right, but because it asks the narrower question (a lone word in both vocabularies), not because of the frame. The converse holds too: the frame counter cannot see C-level work (`x in some_list`, slices, string concatenation), so "no new frames" is not "no new cost" — #553's quadratic was a list membership test inside a loop, invisible to every frame guard, and #1036's a slice assignment shifting the piece list once per merge, which a dev machine's clock read as linear and only CI's runners did not (`_ShiftCountingList` in `tests/v2/test_benchmark.py` counts those shifts now). - **A claim about WHICH STAGE or WHICH LAYER does something is checkable — check it before writing it.** The pipeline is eight stages with a written ownership map (`ParseState`'s docstring, pinned by `tests/v2/pipeline/test_state.py`), and `parse(s).tokens` prints every token's role and tags, so "extract assigns this", "classify never sees that", "group consumes it" each have a one-command answer. #329's prose claimed delimited maiden content is *"claimed whole before classify has tagged anything inside it"*; measured, `classify` tags the marker fine and only the CONSUMING is missing, because `_group`'s rule walks `pieces` and a token that already carries a role is not in `pieces`. Two different mechanisms, one plausible sentence covering both. That single claim then shipped SIX times across three correction rounds, which is the part worth internalizing: **when a mechanism claim turns out wrong, sweep for where else you wrote it, and sweep again at the END of the change over the words the change itself just added.** Prose density here means one idea lives in a docstring, a case note, a release-log entry and this file at once; the implementer working against a wrong mechanism is the person most likely to restate it; and rewriting a mechanism claim is writing one, so the correction earns the same one-command check as the original — two of the six instances were fresh errors introduced by the sentence fixing the previous one. **Adjacency is the trap.** The claim that feels already-known is the one about the neighbouring stage or the neighbouring layer: one comment block in the 2.x `tests/v2/test_facade_cases.py` got the exception type, the raising layer, the skip mechanism, the count of skipped rows, and which row was blocked all wrong at once — every one a claim about `_config_shim` versus real 1.4.0, written from reasoning, in the file whose whole job is translating between them. What finally held was not better prose but moving the claim into a test (`_CORE_ONLY_IDS`, deleted with the facade in 3.0), which cannot be wrong the way a sentence can. - **Typing/docs**: `from __future__ import annotations`; `frozen=True, slots=True` on every public dataclass; strict-profile mypy flags via per-module overrides in pyproject (`strict = true` itself is not valid per-module). Docstrings state contracts in prose with **no doctest blocks** — `--doctest-modules` makes every example a test; behavior examples go to unit tests per the lean-docs rule. **Document the positive direction of a partial property**: "a non-empty `ambiguities` is a signal to act on" is checkable, while "an empty one means no fork occurred" is a universal negative needing exhaustive verification -- that claim was written twice and falsified twice, at sites the author had not audited. @@ -286,7 +289,7 @@ The 2.0 rewrite landed as underscore-private modules beside the v1 code, which 2 **Comparing configurations: use `tools/differential/config_diff.py`, not a throwaway script.** Parsing the corpora under two or more configurations in one tree (the default against a choice, plain against strict, a lexicon with a word removed via `expr:`) is `config_diff.py diff`, parallel, with corpus, country, regex and sample filters, the moves grouped by field and the words whose role changed ranked; `config_diff.py gaps` is the per-language strict-gap report on top of it, and `--readings` writes every name's readings so two trees' runs compare line by line. #1000, #1005, #1017 and #1019 each wrote a single-process script for this (tools/differential/README.md, "Measuring a change"). -**Validating a new parsing rule** — before implementing, simulate it in a throwaway script against `TEST_NAMES` (plus a few target-language examples) to catch regressions/false-positives early. E.g. this surfaced the `last_base` `do`/`st`/`mc` empties and the patronymic `"David Michael Abramovich"` false-reorder. After the change, `tests/v2/test_snapshot.py` lists every corpus name it moved, grouped by changed field -- read them, and re-record with `uv run python tools/differential/snapshot.py --record` once every move is intended, so the PR diff of `snapshot.jsonl` shows the reviewer the same list. `uv run python tools/differential/cldr_agreement.py` reports how the tree's agreement with CLDR's stated name fields moved; `tests/v2/test_cldr_agreement.py` fails on any move, and `tools/differential/cldr_agreement.json` doubles as a measured list of names the parser and CLDR read differently. The same command and test score the GND corpus's real names (`gnd_fields.jsonl`, record `gnd_agreement.json`) by the same rules, and `--record` rewrites every record, printing per source what it accepts. A locale-pack change reads the second view the same command prints, per pack (`cldr_agreement_locales.json`, #672, and `wikidata_native_agreement_locales.json` for real Han and kana names); its `JA` rows run only with the `ja` extra installed. A change to Han, Hangul or kana order or to Korean segmentation reads `wikidata_native_agreement.json` too: the only real names in those scripts the corpora hold. It holds no honorific, so a change to the honorific peel moves nothing there. A change to comma or credential handling also reads the local corpus of hand-typed names (`snapshot.py --local ctgov-officials --base `, which reads the base from a baseline kept for its package's files and never the shared snapshot beside the corpus (a base read once from the same fetch is copied rather than parsed again; `--sample N` reads a fixed sample while iterating; the shared snapshot beside the corpus is one file per source for every worktree, and #752's phase 2f found ctgov reporting a sibling branch's 1,334 joined-credential readings as its own moves against it); tools/differential/README.md's "Local corpora"): the one population of names nobody chose for a test or a catalog, so it shows how often each shape is typed, and the one no CI run sees. A change to title handling reads `ukec-individuals` the same way: UK donors' names in spoken order and mixed case, in 8,219 of whose 15,214 `parse()` read a title on 2026-10-22. A change to English titles, honorifics or post-nominals also reads `snapshot.py --local govuk-people` the same way: GOV.UK's display names of ministers, officials and officers, the title run and the post-nominals as an office writes them (`The Rt Hon`, `Sir`, `Baroness`, ranks; `KC`, `CBE`, `KCB`), in 2,093 of whose 6,021 `parse()` read one or both on 2026-10-30. A change to Spanish titles, to the `es` pack, or to how a title marks where a name starts after an office reads `snapshot.py --local dof-signatures` the same way: the names in the signature blocks of Mexico's Diario Oficial for 2025, cut from the clause that names the signatory's office, 65.4% of its 33,565 signatures opening with a title (`Lic.`, `Dr.`, `Mtro.`, `C.`, `Licenciada`, `C.P.`), whose first word `parse()` read as a title in 82.1% and `parser_for(locales.ES)` in 86.8% on 2026-10-31. A change to particles or generational words reads `snapshot.py --local tse-donors` the same way, Brazilian names in spoken order dense with both. A change to initials, to a title word outside English, or to how a script's combining marks are read reads `snapshot.py --local nepal-candidates` the same way: Nepal's local election candidates, each a romanized name (surnames written as initials, `B.K.`, `K.C.`; honorific titles such as `Guru` in front, `Shekh`, a title there until #752's phase 2j made it one only inside a title phrase, and `Md.`, which its period makes a title by shape) beside the same name in Devanagari, written with vowel signs and the virama, combining marks the letter-shape tests do not count as letters (decisions/nfd-shapes), and with the joiners ZWJ and ZWNJ. It is the only large population of South Asian names the corpora hold, and its corpus reads both scripts. A change to non-Latin scripts or to Arabic-script names reads `snapshot.py --local jrc-names` the same way, and `local_corpora.py groups jrc-names`: the spellings of one person across scripts and transliterations, harvested from news text, with 17,074 Arabic-script spellings where the committed corpora hold 315 distinct names with an Arabic letter, 96 of them CLDR's fictional renderings (2026-10-24), and under a licence that keeps even a reading of one of its names in the cache. A change to which field a word lands in also reads `local_agreement.py psc-individuals --base ` the same way: real names with the four boxes their filers typed them into (UK companies' people with significant control), scored field by field and by nationality; and `local_agreement.py orcid-researchers --base `, ORCID holders' own two boxes (given names, family name) from every country, scored by the country they work or live in -- a population each person split for themselves, academic but worldwide, the one to read for a change a language pack makes; and `local_agreement.py patentsview-inventors --base `, US patents' inventors in the two boxes their filings give, split by a professional filer, with how many people filed under each name in each country, the one to read for how common a name's word is among a country's US-patent inventors. A change to titles or post-nominals, or to the `cs` pack, reads `local_agreement.py cz-candidates --base `, and with `--locale cs` too: Czech election candidates' four boxes (the title before the name, given names, surname, the title after it), scored by election, the only population whose titles are stated apart from the name, 36% of its 170,181 records titled (2026-10-30), nearly all academic degrees; its names are composed from the boxes (`Ing. Jan Novák, Ph.D.`), so it says nothing about how a writer spaces or punctuates them. A change to family-first order, to Vietnamese names or to their titles reads `local_agreement.py uit-vinames --base ` and `local_agreement.py vn-assembly --base `, and with `--locale` once a pack reads Vietnamese (#1108): 21,090 distinct names and 499 National Assembly deputies written family name first with their diacritics, whose parts no one filed but the convention derives (first word the family, last the given, the words between the middle name), so they score middle names, which ORCID's two boxes cannot, and `vn-assembly` the `GS.`/`PGS.`/`TS.`/`ThS.` titles composed from the education each deputy states. These are the only real fielded populations outside the committed fields files, and no CI run sees them either. +**Validating a new parsing rule** — before implementing, simulate it in a throwaway script against `TEST_NAMES` (plus a few target-language examples) to catch regressions/false-positives early. E.g. this surfaced the `last_base` `do`/`st`/`mc` empties and the patronymic `"David Michael Abramovich"` false-reorder. After the change, `tests/v2/test_snapshot.py` lists every corpus name it moved, grouped by changed field -- read them, and re-record with `uv run python tools/differential/snapshot.py --record` once every move is intended, so the PR diff of `snapshot.jsonl` shows the reviewer the same list. `uv run python tools/differential/cldr_agreement.py` reports how the tree's agreement with CLDR's stated name fields moved; `tests/v2/test_cldr_agreement.py` fails on any move, and `tools/differential/cldr_agreement.json` doubles as a measured list of names the parser and CLDR read differently. The same command and test score the GND corpus's real names (`gnd_fields.jsonl`, record `gnd_agreement.json`) by the same rules, and `--record` rewrites every record, printing per source what it accepts. A locale-pack change reads the second view the same command prints, per pack (`cldr_agreement_locales.json`, #672, and `wikidata_native_agreement_locales.json` for real Han and kana names); its `JA` rows run only with the `ja` extra installed. A change to Han, Hangul or kana order or to Korean segmentation reads `wikidata_native_agreement.json` too: the only real names in those scripts the corpora hold. It holds no honorific, so a change to the honorific peel moves nothing there. A change to comma or credential handling also reads the local corpus of hand-typed names (`snapshot.py --local ctgov-officials --base `, which reads the base from a baseline kept for its package's files and never the shared snapshot beside the corpus (a base read once from the same fetch is copied rather than parsed again; `--sample N` reads a fixed sample while iterating; the shared snapshot beside the corpus is one file per source for every worktree, and #752's phase 2f found ctgov reporting a sibling branch's 1,334 joined-credential readings as its own moves against it); tools/differential/README.md's "Local corpora"): the one population of names nobody chose for a test or a catalog, so it shows how often each shape is typed, and the one no CI run sees. A change to title handling reads `ukec-individuals` the same way: UK donors' names in spoken order and mixed case, in 8,219 of whose 15,214 `parse()` read a title on 2026-10-22. A change to English titles, honorifics or post-nominals also reads `snapshot.py --local govuk-people` the same way: GOV.UK's display names of ministers, officials and officers, the title run and the post-nominals as an office writes them (`The Rt Hon`, `Sir`, `Baroness`, ranks; `KC`, `CBE`, `KCB`), in 2,093 of whose 6,021 `parse()` read one or both on 2026-10-30. A change to Spanish titles, to the `es` pack, or to how a title marks where a name starts after an office reads `snapshot.py --local dof-signatures` the same way: the names in the signature blocks of Mexico's Diario Oficial for 2025, cut from the clause that names the signatory's office, 65.4% of its 33,565 signatures opening with a title (`Lic.`, `Dr.`, `Mtro.`, `C.`, `Licenciada`, `C.P.`), whose first word `parse()` read as a title in 82.1% and `parser_for(locales.ES)` in 86.8% on 2026-10-31. A change to particles or generational words reads `snapshot.py --local tse-donors` the same way, Brazilian names in spoken order dense with both. A change to initials, to a title word outside English, or to how a script's combining marks are read reads `snapshot.py --local nepal-candidates` the same way: Nepal's local election candidates, each a romanized name (surnames written as initials, `B.K.`, `K.C.`; honorific titles such as `Guru` in front, `Shekh`, a title there until #752's phase 2j made it one only inside a title phrase, and `Md.`, which its period makes a title by shape) beside the same name in Devanagari, written with vowel signs and the virama, combining marks the letter-shape tests do not count as letters (decisions/nfd-shapes), and with the joiners ZWJ and ZWNJ. It is the only large population of South Asian names the corpora hold, and its corpus reads both scripts. A change to non-Latin scripts or to Arabic-script names reads `snapshot.py --local jrc-names` the same way, and `local_corpora.py groups jrc-names`: the spellings of one person across scripts and transliterations, harvested from news text, with 17,074 Arabic-script spellings where the committed corpora hold 315 distinct names with an Arabic letter, 96 of them CLDR's fictional renderings (2026-10-24), and under a licence that keeps even a reading of one of its names in the cache. A change to which field a word lands in also reads `local_agreement.py psc-individuals --base ` the same way: real names with the four boxes their filers typed them into (UK companies' people with significant control), scored field by field and by nationality; and `local_agreement.py orcid-researchers --base `, ORCID holders' own two boxes (given names, family name) from every country, scored by the country they work or live in -- a population each person split for themselves, academic but worldwide, the one to read for a change a language pack makes; and `local_agreement.py patentsview-inventors --base `, US patents' inventors in the two boxes their filings give, split by a professional filer, with how many people filed under each name in each country, the one to read for how common a name's word is among a country's US-patent inventors. A change to titles or post-nominals, or to the `cs` pack, reads `local_agreement.py cz-candidates --base `, and with `--locale cs` too: Czech election candidates' four boxes (the title before the name, given names, surname, the title after it), scored by election, the only population whose titles are stated apart from the name, 36% of its 170,181 records titled (2026-10-30), nearly all academic degrees; its names are composed from the boxes (`Ing. Jan Novák, Ph.D.`), so it says nothing about how a writer spaces or punctuates them. A change to family-first order, to Vietnamese names or to their titles reads `local_agreement.py uit-vinames --base ` and `local_agreement.py vn-assembly --base `, and with `--locale vi`, `locales.VI` being that pack (#1108): 21,090 distinct names and 499 National Assembly deputies written family name first with their diacritics, whose parts no one filed but the convention derives (first word the family, last the given, the words between the middle name), so they score middle names, which ORCID's two boxes cannot, and `vn-assembly` the `GS.`/`PGS.`/`TS.`/`ThS.` titles composed from the education each deputy states. These are the only real fielded populations outside the committed fields files, and no CI run sees them either. ## Gotchas diff --git a/docs/concepts.rst b/docs/concepts.rst index 855f11803..72670711e 100644 --- a/docs/concepts.rst +++ b/docs/concepts.rst @@ -77,7 +77,11 @@ and ``script_orders`` change what "first" and "last" mean here, as does the opt-in ``two_family_names``, which takes the last two name words as the family (:ref:`two-family-names`), and the opt-in ``two_given_names``, which gives the given name the middle name beside -it (:ref:`two-given-names`). +it (:ref:`two-given-names`). A name of one name word has no first and +last to tell apart: it is the given name under every ``name_order``, or the +family name where ``lone_name_word`` says so, where a title beside it +makes it one (``"Mr. Johnson"`` is family ``Johnson``), or where its +script settles the order (``"王"``) (:ref:`lone-name-word`). ``script_orders`` covers the East Asian writing that settles its own order: a name wholly in Han or hangul, or a Japanese name in kanji and kana — any of them, hiragana alone included — unless it is katakana @@ -85,7 +89,7 @@ alone, which is how a foreign name is transcribed. The Chinese interpunct ``·`` dividing such a name walks that back, marking a transcription that follows ``name_order`` like any other name. Three rules can still override the arrangement: a lone name word beside a given-name title -such as ``Sir`` is the given name under every order, a particle that can never be a given name, +such as ``Sir`` is the given name whatever ``lone_name_word`` says, a particle that can never be a given name, opening the name, opens the surname under every order rather than becoming the given name (both in :ref:`order-vocabulary-first`), and the opt-in ``patronymic_rules`` reorder, under the default order, a name diff --git a/docs/customize.rst b/docs/customize.rst index 82cc361de..a42e4cd6a 100644 --- a/docs/customize.rst +++ b/docs/customize.rst @@ -854,7 +854,8 @@ listed below. ``False``. * - ``two_family_names`` - ``bool`` - - Reads the last two name words as the family name, for data + - Reads the last two name words as the family name (the first + two under a family-first order), for data where people carry two family names (Spain, most of Latin America). Defaults to ``False``. See :ref:`two-family-names`. * - ``two_given_names`` @@ -1182,8 +1183,9 @@ holds family names. For data whose lone words are surnames, set Anything that does decide the word still decides it first, whatever the switch says: a title (``"Mr. Johnson"`` is family ``Johnson``, ``"Sir -John"`` given ``John``), a nickname or maiden name beside it, and a -Han or Hangul name, which its script reads family-first. Through 2.x +John"`` given ``John``), a nickname or maiden name beside it, a family +comma (``"Anh, Jr."`` is family ``Anh``, ``", Anh"`` given ``Anh``), and a name its script reads family-first: Han, Hangul, or kana other than +katakana alone (``"みなみ"`` is family ``みなみ``). Through 2.x the field followed ``name_order``, so a family-first order read a lone word as the family name. diff --git a/docs/design/decisions/vocabulary-model.md b/docs/design/decisions/vocabulary-model.md index 83f7430f3..3033228c7 100644 --- a/docs/design/decisions/vocabulary-model.md +++ b/docs/design/decisions/vocabulary-model.md @@ -586,7 +586,7 @@ Open: #753 (phase 3, language packs; the 16 words strict ungates, each read for - **2026-11-05 #1108 (its `default = false` on `ths`, `đbqh` and `dbqh` superseded by the 2026-11-06 #1115, the pack titles entry; `ts`, `gs` and `pgs` stay the choice's, written as initials too; its accepted stacked title ending a name, given 'PGS.TS.', superseded by decisions/H5's 2026-11-06 #1116 entry, which reads it as the title; its accepted lone name word read as the family under the pack, `Anh` → family 'Anh', superseded by decisions/O5's 2026-11-06 #1111 entry: a lone word reads given under every order, the pack's included) — `locales.VI`: A PLAIN CHOICE OF `vi`, vi.toml'S `[rules]` ORDER (`name_order = "family-first-given-last"`), AND A LEXICON FRAGMENT THAT REMOVES THE READINGS OTHER FILES GIVE THE SYLLABLES VIETNAMESE NAMES ARE WRITTEN WITH: PARTICLE (`van`, `le`, `do`, `du`, `di`, `la`, `tho`, `da`, `san`), CONNECTIVE (`the`, `y`) AND CREDENTIAL (`do`, `ba`, `chi`); THE ONE PACK FRAGMENT THAT REMOVES; NO STRICT MODE; THE DEFAULT DOES NOT MOVE.** Decided in chat on 2026-11-05 ("i agree with route A"; the word list "all as recommended"; the credentials "yes, send that", and `ma` left out, confirmed in chat), over (B) a pack of the order and titles alone, documented with `strict=True`, and (C) a `strict` field on `Locale`. #1108 proposed `given` readings of `văn`/`van` and `thị`/`thi` in vi.toml, on the premise that a reading turned on also prefers. It does not: `choose` prefers only by a vocabulary reading, and a name reading never settles (decision 15), so under a plain choice `Nguyen Van Anh` read given 'Van Anh', the Dutch particle chaining `Van` into the given name, exactly as the order alone reads it. No kind a file can write would make `van` a name word, so the settling is done by the pack's fragment, whose `remove` calls apply after the choice as a caller's own do. It bends decisions 6 and 15, which bind the default and the packs, for this pack alone, on the ground that its order commits a caller to Vietnamese names: a name written without a comma that the removal could misread (`Ludwig van Beethoven`) is misread by the order first. A family comma fixes the order, and the removals still act there, a cost the pack adds on its own, accepted below. `mechanisms.md#LOCALE-PACKS-PURE-DATA`'s contract names the exception. No compiled set of the default moves (`tools/vocab/build.py` reports `_compiled.py` current), and `snapshot.jsonl` is unchanged. - **The measurement**, on the local corpora #1110 added (`tools/differential/local_uit.py`, UIT-ViNames; `tools/differential/local_vnassembly.py`, the National Assembly deputies), each record's parts derived from its order (the first word the family, the last the given, the words between the middle), measured 2026-11-05 and re-taken on the commit adding the credential removal, which moved none of them (default / the order alone / VI / strict VI): UIT-ViNames, 21,090 names written with diacritics, 13 / 21,065 / 21,085 / 21,089; the same names with the marks stripped (đ as d), standing in for romanized writing, 89 / 19,221 / 21,082 / 21,089 (the order to VI +1,861/−0); the deputies, 499 with the titles composed from their education, 0 / 483 / 499 / 499, stripped 0 / 404 / 499 / 499 (the order to VI +16, the stacked `PGS.TS.` and `GS.TS.`, and +95 stripped). The Vietnamese nationals of `psc-individuals`, filed in Western order, are the contrast: 1,097 of 1,494 by default and 1 under the order, the pack adding no loss beyond the order's. Per word, the records VI loses with the word left out of the fragment (UIT stripped / deputies stripped): `van` 1,221 / 74, `le` 398 / 2, `the` 163 / 3, `do` 40 / 0, `du` 15, `da` 11, `y` 10, `di` 9, `la` 3, `san` 3 (and 3 written with diacritics), `tho` 2 / 1; with diacritics only `du` 7, `di` 4, `la` 3, `san` 3, `da` 2 and `the` 1 fire. `mac`, in the first list, moved no Vietnamese-order name in any corpus (2 PSC Western-order names, losing nothing) and was dropped; `san`, the saint particle chaining a given name (`… San Hà`), was added (+3/−0 either way). Earlier and not re-taken: ORCID's Vietnam records composed family-first, 43,790 with both boxes, counting a middle the filer put in the family box as agreeing, read 89.95% under the order, 94.72% with the first fragment (`mac` in, `san` out), 95.43% strict. - **The credentials**, removed after the code review found `NGUYEN VAN DO` reading suffix 'DO' under the pack: removing `van`'s particle had left en.toml's credential `DO` to take the given name of a name in one case, and `BA` and `CHI` did so by default. The comparator is VI with the credential readings kept. GAINED: the removal moves a role only in a name written in capitals, the syllable read as the given name where the credential took it: UIT-ViNames stripped and in capitals 108 names (the recipe prints 109 moved, one moving only its report, `KA BA`) (`chi` 82, `do` 20, `ba` 6), in capitals with its diacritics 68 (`chi` 65, `ba` 3), the deputies in capitals 4 (`chi` 2, `ba` 2), every one a gain by the corpora's own parts, no loss; as written and stripped in mixed case it moves only the `suffix-or-name` report (69 UIT names and 4 deputies' names as written, 110 and 4 stripped). FEC's donors with a common Vietnamese surname gain too, a given name the default reads as a credential: `chi` 5 (`NGUYEN, VY CHI` → middle 'CHI'), `ba` 2. COST: among names with a common Vietnamese surname (1,200 in ctgov-officials, 3,294 in FEC), a real credential stops being a suffix in `do` 4 (`Ryan Nguyen, DO`, `Thanh D Hoang, DO`, `Thao L Nguyen, DO, FAAP`; FEC `NGUYEN, MINH QUANG LCDR DO, MPH,`) and `ba` 1 (`Phuong T Vo, BA`), `chi` none. Every ctgov one is written given-first for English-language readers, which VI misreads with the credential kept (`Ryan Nguyen, DO` → family 'Ryan') and the default reads right. Measured 2026-11-05. Recompute the gains with `uv run python tools/differential/config_diff.py diff 'expr:parser_for(Locale(code="vi", lexicon=Lexicon.empty().remove("da","di","do","du","la","le","san","tho","van", kind="particle").remove("the","y", kind="conjunction"), policy=locales.VI.policy, languages=locales.VI.languages))' plain:VI --local uit-vinames --spelling stripped,upper` (`--spelling` added for this; `upper` alone, or `--local vn-assembly`, for the other rows); the costs by the same two configurations over the ctgov and FEC names holding one of `STRONG`'s surnames (the session notes' `credcost.py`, not in a tree). - - **Accepted for the credentials:** `DO` or `BA` written as a credential after a Vietnamese name reads as a name word, with a comma or without, dotted (reported `suffix-or-name`), in brackets or behind another credential (`Nguyen Van Anh, DO` → family 'Nguyen Van Anh', given 'DO'; `Nguyen Van Anh DO`, `Nguyen Van Anh D.O.` → given 'DO'; `Nguyen Van Anh (DO)` → nickname 'DO'; `Nguyen Van Anh MD, DO` → given 'DO'); past a second comma it stays a suffix (`Nguyen, Van Anh, DO`). `CHI` reads so without a comma (`Nguyen Van Anh CHI` → given 'CHI'); after one the capitals shape (rules/C6) takes a three-letter word in capitals as a suffix whatever its readings, so `Nguyen Van Anh, CHI` keeps suffix 'CHI', reported. Left out: `ma`, which gained no corpus name and cost one (`Linda Trinh, MA`), so `TRAN THI MA` stays a known gap (suffix 'MA'). Not removable: `VI`, a given name read as the numeral by its shape (`NGUYEN THI VI`). Declined for 3.0: keeping the credential readings and declining only the bare word ending the core of a one-case name, which would keep the comma costs away but loses FEC's family-comma gains, and needs a new `Policy` field set from vi.toml's `[rules]` (packs carry no code, and `leading_only`/`trailing_only` are reserved, unbuilt, for 3.x), amending rules/S4 (an ambiguous acronym needs words to spare) at 3.0 final. + - **Accepted for the credentials:** `DO` or `BA` written as a credential after a Vietnamese name reads as a name word, with a comma or without, dotted (reported `suffix-or-name`), in brackets or behind another credential (`Nguyen Van Anh, DO` → family 'Nguyen Van Anh', given 'DO'; `Nguyen Van Anh DO`, `Nguyen Van Anh D.O.` → given 'DO'; `Nguyen Van Anh (DO)` → nickname 'DO'; `Nguyen Van Anh MD, DO` → given 'DO'); past a second comma it stays a suffix (`Nguyen, Van Anh, DO`). `CHI` reads so without a comma (`Nguyen Van Anh CHI` → given 'CHI'); in the part right after one, the capitals shape (rules/C6) takes an unlisted word of three letters or more in capitals as a suffix behind two name words or more (C4) where the name's case contrasts, so `Nguyen Van Anh, CHI` keeps suffix 'CHI', reported, while a name written wholly in capitals gives the shape nothing to contrast with (`NGUYEN VAN ANH, CHI` → given 'CHI', unreported), and one name word before the comma leaves it nothing to spare (`Anh, CHI` → given 'CHI'); past a second comma it stays a suffix in any case, as `DO` does (rules/C2: `NGUYEN, VAN ANH, CHI` → suffix 'CHI'). (Corrected 2026-11-07: the entry said the shape takes the word whatever its readings, after any comma; it takes `CHI` only because the pack removes its credential reading, and only where the name's case contrasts, as C6 states.) Left out: `ma`, which gained no corpus name and cost one (`Linda Trinh, MA`), so `TRAN THI MA` stays a known gap (suffix 'MA'). Not removable: `VI`, a given name read as the numeral by its shape (`NGUYEN THI VI`). Declined for 3.0: keeping the credential readings and declining only the bare word ending the core of a one-case name, which would keep the comma costs away but loses FEC's family-comma gains, and needs a new `Policy` field set from vi.toml's `[rules]` (packs carry no code, and `leading_only`/`trailing_only` are reserved, unbuilt, for 3.x), amending rules/S4 (an ambiguous acronym needs words to spare) at 3.0 final. - **Accepted for comma names:** after a family comma, where the order reads nothing differently, the removals still act, so a particle or a credential there reads as a name word: `Beethoven, Ludwig van` → family 'Beethoven', middle 'van', where the default gives family 'van Beethoven'; `Smith, John Do` → middle 'Do'. UIT-ViNames and the deputies hold no comma names, so no corpus measures it. - **Recompute the four-parser table** with `local_agreement.read` over the uit-vinames and vn-assembly records under `Parser()`, `Parser(policy=Policy(name_order=FAMILY_FIRST_GIVEN_LAST))`, `parser_for(locales.VI)` and `parser_for(locales.VI, strict=True)` (the session notes' `vi_measure.py`, not in a tree; `local_agreement.py --locale vi --base` scores the base under the pack too, so it cannot give the default against it); `config_diff.py diff default plain:VI --local uit-vinames`, with `--spelling stripped` for the romanized view, gives the moves, and an `expr:` fragment leaving one word out each word's. - **The titles.** vi.toml turns on, `default = false`, the academic titles written before a name (`ts`, `ths`, `gs`, `pgs`) and the office `đbqh` (đại biểu Quốc hội, National Assembly deputy) with its ASCII spelling `dbqh`. A dotted title is one at the head by shape already (rules/H2), so a reading is what reads the bare form (`Ts Nguyen Van Anh`) and the stacked one (`PGS.TS.`, each chunk a listed title, `PGS.TS.BS.` included, `bs` being en.toml's credential): the deputies' composed titles are ThS. 241, TS. 142, PGS.TS. 9 and GS.TS. 7, and the 16 stacked ones are exactly what the order alone misses. None of the four has a Vietnamese bearer (tools/name_usage.py, the counts on each word). `đbqh` is in none of the three corpora; CLDR 48's vi file writes it before a surname-first name (`ĐBQH Nguyễn Văn Anh`, which now reads family 'Nguyễn'). The office makes title phrases with `chief` and `deputy`, which attach to every office, so both join the choice's derived surface; neither reads differently. Accepted: a stacked title ending a name reads as the given name (`Nguyen Van Anh PGS.TS.` → given 'PGS.TS.'), where the default reads suffix 'PGS.TS.' by rules/S3's unlisted dotted suffix, which no longer takes it once `pgs` and `ts` are listed, and rules/H5 takes only single words; no local or committed corpus writes one trailing (all 16 `PGS.TS.` and `GS.TS.` lead, in vn-assembly, measured 2026-11-05), and keeping the readings to the leading position waits for `leading_only`, reserved for 3.x. Declined: `bs` (bác sĩ, physician), whose title reading, preferred under the choice, evicted en.toml's credential `BS`, so `Smith, John BS` read middle 'BS'; excluded in vi.toml with that reason. Not proposed: the military ranks, which the deputies' data writes in the position, never the name, and the monks' titles (Hòa thượng, Thượng tọa), full words found only in aliases. diff --git a/docs/design/rules/O4-positional-assignment-and-declared-order.md b/docs/design/rules/O4-positional-assignment-and-declared-order.md index 44218fb6c..b1693377a 100644 --- a/docs/design/rules/O4-positional-assignment-and-declared-order.md +++ b/docs/design/rules/O4-positional-assignment-and-declared-order.md @@ -6,6 +6,7 @@ - **In the default given-first order** the first name word is the given name, the last is the family name, and everything between is middle names. - **In a family-first order** the first name word is the family; in family-first-given-last the given name comes from the end, the middles from between. +- **A name of one name word** is O5's under every order. **Because:** what no vocabulary claims can only be read by where it stands, under the order the caller declared. diff --git a/docs/design/rules/S3-credentials-written-in-joined-chunks.md b/docs/design/rules/S3-credentials-written-in-joined-chunks.md index 4884bd972..e3a9bc40a 100644 --- a/docs/design/rules/S3-credentials-written-in-joined-chunks.md +++ b/docs/design/rules/S3-credentials-written-in-joined-chunks.md @@ -68,7 +68,7 @@ "Jane Smith OTR/L/X" → family="OTR/L/X" · boundary "Jane Smith-NP-C" → family="Smith-NP-C" · boundary ``` -- **A member of the ambiguous class (S4) is a chunk too, however it is written, where the word holds a chunk that settles it**: a credential that is no name, or a member written with a period inside it, and not a generation, a numeral, an honorific or a word that is a title too, which say nothing about the member, as in a clause (S1), nor a member's case, which a chunk cannot be read by. Written alone, such a member may be a name or the initials it looks like (`John Smith MA` and `John Smith M.A` keep the family), but a credential joined to it says what the whole word is, since nobody joins a name or initials to a credential. A word of bare members alone has nothing to settle it, and a name word beside a member still refuses the word. +- **A member of the ambiguous class (S4) is a chunk too, however it is written, where the word holds a chunk that settles it**: a credential that is no name, or a member written with a period inside it, and not a generation, a numeral, an honorific or a word that is a title too, which say nothing about the member, as in a clause (S1), nor a member's case, which a chunk cannot be read by. Written alone, such a member may be a name or the initials it looks like (`John Smith Ma` and `John Smith M.A` keep the family), but a credential joined to it says what the whole word is, since nobody joins a name or initials to a credential. A word of bare members alone has nothing to settle it, and a name word beside a member still refuses the word. ``` "John Smith MA/PhD" → suffix="MA/PhD" "John Smith M.A/PhD" → suffix="M.A/PhD" diff --git a/docs/locales.rst b/docs/locales.rst index 181ff625b..ecf0aab6f 100644 --- a/docs/locales.rst +++ b/docs/locales.rst @@ -1083,8 +1083,10 @@ name words are name words there too (``Beethoven, Ludwig van`` gives middle ``van``). ``DO`` or ``BA`` written after a Vietnamese name as a credential, for English-language readers, reads as a name word, with a comma or without (``Nguyen Van Anh, DO`` gives family ``Nguyen Van -Anh``, given ``DO``), and ``CHI`` without a comma; other credentials, -and ``CHI`` after a comma, stay suffixes (``Nguyen Van Anh, PhD``). +Anh``, given ``DO``), and ``CHI`` without a comma or in a name written +wholly in capitals (``NGUYEN VAN ANH, CHI`` gives given ``CHI``); other +credentials, and ``CHI`` after a comma in a name written in mixed case, +stay suffixes (``Nguyen Van Anh, PhD``). .. _segmenter-contract: diff --git a/docs/migrate.rst b/docs/migrate.rst index 7638b337d..2df4ea6f4 100644 --- a/docs/migrate.rst +++ b/docs/migrate.rst @@ -475,7 +475,7 @@ not ``suffix_acronyms_ambiguous``, say) is refused, naming the fields; so are a ``Parser`` and a ``Locale`` holding it. A loaded lexicon keeps the vocabulary it was pickled with, not 3.0's: a 2.8.0 pickle of ``Lexicon.default()``, or of a ``Parser()``, loads as the 3.0 default -with the 23 credentials and 13 case masks 3.0 added taken out again, so it reads +with the credentials and case masks 3.0 added taken out again, so it reads ``"Jane Doe KC"`` as 2.8.0 did. Build a new ``Parser()`` for 3.0's vocabulary. @@ -1208,6 +1208,31 @@ with what 1.4 read: MACHADO"`` — 1.4 wrote ``st``, ``dela`` and ``MacHado``; saint and rank particles keep their capital, and a name opening Mac is title-cased unless the vocabulary records its casing (``MacDonald``). +- ``"John Smith y V"`` — a lone letter after a connective is part of + the family: 1.4 read middle ``Smith``, last ``y``, suffix ``V``. +- ``"이서연·김민준"`` typed decomposed (NFD) — the interpunct divides + it as it does the composed spelling: 1.4 read first + ``이서연·김민준``. +- ``"Hans Meier Priv.-Doz."`` — a listed title with a period or hyphen + inside it ending a name is a title: 1.4 read middle ``Meier``, last + ``Priv.-Doz.``. +- ``"Jane Smith PhD/NP-C"`` and ``"John Smith MA/PhD"`` — a joined + credential is the suffix, a credential listed with its own joiner and + a credential that is also a name beside one that is not included: 1.4 + read middle ``Smith``, last ``PhD/NP-C`` and ``MA/PhD``. +- ``"John Smith (Ma, PhD)"`` and ``"John Smith (Msc.Ed)"`` — 1.4 read + nickname ``Ma, PhD`` and ``Msc.Ed``. +- ``"'Smitty' Jones Jr."`` — a nickname, one name word and a suffix read + the word as the family: 1.4 read first ``Jones``, last ``Jr.``. +- ``"Mohd. Shameem Khan"`` — Muhammad, Achmad, Francisco and Sheikh + written short are given names: 1.4 read title ``Mohd.``, first + ``Shameem``. +- ``"Jan Novák, MUDr"``, ``"Licenciado Juan Pérez"``, ``"Esra Oz, Assist + Prof"``, ``"Engr Ahsen Ali"`` and ``"Anselm Crombach, Dipl.-Psych."`` + — titles no name bears are titles: 1.4 read first ``MUDr``, + ``Licenciado``, ``Assist``, ``Engr`` and ``Dipl.-Psych.``. +- ``"Mary Smith, Mst"`` — ``Mst`` after a name is a credential: 1.4 + read first ``Mst``, last ``Mary Smith``. .. doctest:: @@ -1241,3 +1266,25 @@ with what 1.4 read: >>> [str(parse(n).capitalized()) for n in ("JOHN ST CLAIR", ... "JUAN DELA CRUZ", "JOHN MACHADO", "RONALD MACDONALD")] ['John St Clair', 'Juan Dela Cruz', 'John Machado', 'Ronald MacDonald'] + >>> parse("John Smith y V").family + 'Smith y V' + >>> import unicodedata + >>> nfd = parse(unicodedata.normalize("NFD", "이서연·김민준")) + >>> nfd.given == unicodedata.normalize("NFD", "이서연") + True + >>> parse("Hans Meier Priv.-Doz.").title + 'Priv.-Doz.' + >>> parse("Jane Smith PhD/NP-C").suffix, parse("John Smith MA/PhD").suffix + ('PhD/NP-C', 'MA/PhD') + >>> parse("John Smith (Ma, PhD)").suffix, parse("John Smith (Msc.Ed)").suffix + ('Ma, PhD', 'Msc.Ed') + >>> parse("'Smitty' Jones Jr.").family + 'Jones' + >>> parse("Mohd. Shameem Khan").given + 'Mohd.' + >>> [parse(n).title for n in ("Jan Novák, MUDr", "Licenciado Juan Pérez", + ... "Esra Oz, Assist Prof", "Engr Ahsen Ali", + ... "Anselm Crombach, Dipl.-Psych.")] + ['MUDr', 'Licenciado', 'Assist Prof', 'Engr', 'Dipl.-Psych.'] + >>> parse("Mary Smith, Mst").suffix + 'Mst' diff --git a/docs/release_log.rst b/docs/release_log.rst index e7b5723f1..5843e61d2 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -42,7 +42,7 @@ Release Log **Behavior Changes** - - **Change a name of one word to read as the given name under every order, with Policy.lone_name_word to choose the family.** ``Parser(policy=Policy(name_order=FAMILY_FIRST)).parse("Anh")`` gives given ``Anh``, where 2.0.0 through 2.8.2 gave family ``Anh``, the field following the order; ``parser_for(locales.VI)`` reads it the same way. For data whose lone words are surnames, ``Policy(lone_name_word=Role.FAMILY)`` reads them as the family under every order, and ``python -m nameparser`` gains ``--lone-name-word``. A title, a nickname, a maiden name or a Han or Hangul script still decides first. :issue:`1111` + - **Change a name of one word to read as the given name under every order, with Policy.lone_name_word to choose the family.** ``Parser(policy=Policy(name_order=FAMILY_FIRST)).parse("Anh")`` gives given ``Anh``, where 2.0.0 through 2.8.2 gave family ``Anh``, the field following the order; ``parser_for(locales.VI)`` reads it the same way. For data whose lone words are surnames, ``Policy(lone_name_word=Role.FAMILY)`` reads them as the family under every order, and ``python -m nameparser`` gains ``--lone-name-word``. A title, a nickname, a maiden name, a family comma or a script that settles the order (Han, Hangul, kana other than katakana alone) still decides first. :issue:`1111` - **Read عبد and a name of God after it as one name in Arabic script, past the first name word.** ``parse("طارق عبد العزيز")`` gives family ``عبد العزيز``, where 1.4.0 through 2.8.1 gave middle ``عبد``, family ``العزيز``; it joins a listed name of God or devotional name after it, and before any other word it stays a name of its own (``نوال عبد هاشم``, ``محمد عبد القريشي``). The Latin ``Abdul`` is unchanged (``Tariq Abdul Aziz`` keeps middle ``Abdul``). :issue:`694` - **Read a lowercase particle opening a family-first name as the start of the surname.** ``Parser(policy=Policy(name_order=FAMILY_FIRST)).parse("van den Berg Cornelia")`` gives family ``van den Berg``, given ``Cornelia``, where 2.0.0 through 2.8.1 gave family ``van``, given ``den Berg Cornelia``, and reports nothing. A capitalized particle, or one in a name written in one case, keeps its reading (``Le Van Thanh`` gives family ``Le``), and the default order is unchanged. :issue:`670` - **Fix a family-first name behind St, Freiherr or Graf taking the given name into the family.** ``Parser(policy=Policy(name_order=FAMILY_FIRST)).parse("St de Mesnil Jean")`` gives family ``de Mesnil``, given ``Jean``, as ``Dr. de Mesnil Jean`` reads, where 2.0.0 through 2.8.1 gave family ``de Mesnil Jean``. :issue:`828` diff --git a/docs/usage.rst b/docs/usage.rst index f0cb78c27..ea5d7b93a 100644 --- a/docs/usage.rst +++ b/docs/usage.rst @@ -1593,6 +1593,15 @@ fixes the order whatever the flag: $ python -m nameparser --order family-first --json "Kovács János Péter" {"title": "", "given": "János", "middle": "Péter", "family": "Kovács", "suffix": "", "nickname": "", "maiden": ""} +``--lone-name-word`` takes ``given`` (the default) or ``family``, the +field a name of one name word reads as under every order +(:ref:`lone-name-word`): + +:: + + $ python -m nameparser --lone-name-word family --json "Anh" + {"title": "", "given": "", "middle": "", "family": "Anh", "suffix": "", "nickname": "", "maiden": ""} + The ``Policy`` switches that turn a reading on or off have a flag each, spelled as the field: ``--two-given-names``, ``--two-family-names``, ``--middle-as-family``, ``--lenient-comma-suffixes`` and diff --git a/nameparser/__main__.py b/nameparser/__main__.py index 93073b623..d4f5c79c4 100644 --- a/nameparser/__main__.py +++ b/nameparser/__main__.py @@ -45,8 +45,9 @@ # tables, --order or _NOT_ON_THE_COMMAND_LINE. _SWITCHES = { "two_given_names": "join a middle name to the given name", - "two_family_names": "read the last two name words as the family " - "(two family names)", + "two_family_names": "read two name words as the family (two " + "family names): the last two, or the first two " + "under a family-first --order", "middle_as_family": "fold the middle names into the family", "lenient_comma_suffixes": "read a lone V or I after a comma as a " "numeral suffix (John Smith, V)", diff --git a/nameparser/_pipeline/AGENTS.md b/nameparser/_pipeline/AGENTS.md index 92bc5c18e..b7fad61aa 100644 --- a/nameparser/_pipeline/AGENTS.md +++ b/nameparser/_pipeline/AGENTS.md @@ -34,7 +34,7 @@ The decision site also has the token index and the detail text in hand, which th What can still do the shifting is narrow, and #367 is why: a plain title no longer can (`Dr. Van Johnson` reads as `Van Johnson` does and reports from `_assign`), so the `_group` emitter needs a word that is BOTH a title and a particle — measured, `TITLES ∩ particles_ambiguous` is `{freiherr, st}` in the default vocabulary, and the ranks `graf`, `gräfin`, `fürst` and `prinz` since #709 (2026-11-04) (`do` left TITLES in #296's audit; the particle-line block in decisions/vocabulary-exclusions records the before and after), plus any overlap a caller's config creates — standing ahead of the chained particle as the LEADING NAME word. Titles may precede it, so `Dr. St van Johnson` reaches the emitter and `St van Johnson` does too; a given name may not, so `Jan Freiherr von Richthofen` does not reach it while `Freiherr von Richthofen` and `Dr. Freiherr von Richthofen` do. Two shapes that look like they should reach it and do NOT, both measured by stepping `STAGES` and watching where `ambiguities` grows: `Dr. Do van Johnson` and `Do St Johnson` report from `assign`, not `group`, because `do` is no longer a title and so stays the leading name piece assign reports on — a both-vocabulary word CHAINED (`Jan St Johnson`) reports nothing at all. When checking whether that emitter is dead, a both-vocabulary word in the leading name position is the thing to look for, and the answer is that it is not dead. -The stage-ownership map in `tests/v2/pipeline/test_state.py` must list `ambiguities` for each such stage, and it passes vacuously until a case row exercises the path, so add the row too. Report BOTH directions of a two-way fork — "John Smith MA" (read as a suffix) and "Jack MA" (read as the family name) are equally guesses. +The stage-ownership map in `tests/v2/pipeline/test_state.py` must list `ambiguities` for each such stage, and it passes vacuously until a case row exercises the path, so add the row too. Report BOTH directions of a two-way fork — "John Smith V" (read as a suffix) and "John V" (read as the family name) are equally guesses. ### Name the reading not taken diff --git a/nameparser/_pipeline/_classify.py b/nameparser/_pipeline/_classify.py index 3bfd33caa..278fbb5c6 100644 --- a/nameparser/_pipeline/_classify.py +++ b/nameparser/_pipeline/_classify.py @@ -47,9 +47,11 @@ "vocab:suffix", "vocab:suffix-word", "vocab:suffix-ambiguous", "vocab:suffix-word-ambiguous", "vocab:agnomen", "vocab:generation-word", "vocab:credential-word", -"vocab:honorific-word", "vocab:patronymic" and "vocab:trailing-title" (read by the later -stages and dropped by assemble, Token.reading carrying the kind out, -save the last, which marks no kind), +"vocab:honorific-word", "vocab:patronymic", "vocab:trailing-title" +and "vocab:joined-title" (read by the later stages and dropped by +assemble, Token.reading carrying the kind out, save the last two, +which mark no kind), "vocab:title-case-name" (a suffix word whose +Title case leans to the name, rules/S6, kept on the published token), "vocab:particle-ambiguous", "vocab:bound-given", "vocab:given-abbreviation", "vocab:shared-name" (which rules/P10, O6 and O7 read), "vocab:particle-given" (which P10 and O6 read) and diff --git a/nameparser/_policy.py b/nameparser/_policy.py index 0e92e355f..b58fc4061 100644 --- a/nameparser/_policy.py +++ b/nameparser/_policy.py @@ -545,7 +545,8 @@ class Policy: only what you change -- ``Policy(maiden_delimiters=frozenset({("(", ")")}))`` -- and pass the result to ``Parser(policy=...)``.""" - #: How positional (no-comma) input maps onto given/middle/family. + #: How positional (no-comma) input maps onto given/middle/family + #: (a name of one name word is placed by lone_name_word instead). #: Valid values are exactly the three exported #: :ref:`name-order constants ` -- #: GIVEN_FIRST (the default), FAMILY_FIRST, and @@ -1067,8 +1068,8 @@ def __post_init__(self) -> None: f"{f.name} must be an iterable, got {value!r}{hint}" ) from None object.__setattr__(self, f.name, frozenset(value)) - # Every other field (the scalars: the bools and - # unlisted_caps_suffixes) DELIBERATELY gets no type check here, + # Every other field (the scalars: the bools, + # unlisted_caps_suffixes and lone_name_word) DELIBERATELY gets no type check here, # unlike name_order and the union fields above: a # PolicyPatch(strip_emoji="off") is constructible, and only raises # once apply_patch runs Policy.__post_init__'s checks. This is diff --git a/nameparser/locales/pt.py b/nameparser/locales/pt.py index 760b7f792..07499da4f 100644 --- a/nameparser/locales/pt.py +++ b/nameparser/locales/pt.py @@ -1,6 +1,9 @@ """The Portuguese locale pack: vocabulary-only -- it chooses the -language ``pt`` and prefers pt.toml's readings -where another language's clash: ``do`` the particle, not the DO +language ``pt``, which turns on pt.toml's ``default = false`` +readings (on 2026-11-07 only ``piedade``, the second half of a +compound given name behind a particle: ``Maria da Piedade Silva`` gives +middle ``da Piedade``, rules/P10, #1118, #1122), and prefers pt.toml's +readings where another language's readings clash: ``do`` the particle, not the DO credential, and ``e`` the connective even in a name written in one case, while ``Sr`` keeps both readings, Senhor in front and the generation behind, as the default reads it (decisions/vocabulary-model, diff --git a/nameparser/locales/vi.py b/nameparser/locales/vi.py index 0650d713f..f0dd7ab3a 100644 --- a/nameparser/locales/vi.py +++ b/nameparser/locales/vi.py @@ -22,7 +22,8 @@ cost to ``DO`` or ``BA`` written as a credential after a Vietnamese name for English-language readers, with or without a comma (``Nguyen Van Anh, DO`` reads family ``Nguyen Van Anh``, given ``DO``), and ``CHI`` -without one (after a comma the capitals shape keeps a three-letter word +without one or in a name written wholly in capitals (after a comma in a +name written in mixed case the capitals shape keeps a three-letter word a suffix). Data sources: the titles' vetting is on the words in vi.toml @@ -37,7 +38,7 @@ holding one of its syllables, and the choice's titles and the fragment's credentials reach comma names too (``Nguyen, Ts Van Anh``, ``Nguyen Van Anh, DO``), so DEVIATES below declares every name holding -a word. The choice's own words are derived +a letter or a digit. The choice's own words are derived (``_reading_index.changed_words``), which the non-interference gate in tests/v2/test_locales.py ors in. """ diff --git a/nameparser/vocab/AGENTS.md b/nameparser/vocab/AGENTS.md index 368c7c2c2..8654d3d24 100644 --- a/nameparser/vocab/AGENTS.md +++ b/nameparser/vocab/AGENTS.md @@ -4,7 +4,7 @@ Loaded when a session reads a vocabulary TOML file. The steps for adding a word ## Step 1: its other readings -First check the word's OTHER readings: `uv run python tools/vocab/query.py --word ` lists every reading and exclusion of it in every file, which is where the overlaps between categories now show. Real overlaps exist: `st`/`freiherr` ∈ `PARTICLES` ∩ `TITLES`, and `do`/`mc`/`vd` ∈ `PARTICLES` ∩ `SUFFIX_ACRONYMS`; `abd` = "ABD" ∈ `SUFFIX_ACRONYMS`; `abu` ∈ `PARTICLES` ∩ `BOUND_GIVEN_NAMES` (position-dependent: leading token → given-name join, mid-name → family-name join). Usually position-dependent and harmless, but can force a guard or an exclusion (the `last_base` all-particles guard; `abd`, kept out of `BOUND_GIVEN_NAMES` until decisions/P5 found the credential collision cost nothing at either end). +First check the word's OTHER readings: `uv run python tools/vocab/query.py --word ` lists every reading and exclusion of it in every file, which is where the overlaps between categories now show. Real overlaps exist: `st`/`freiherr` and the ranks `graf`, `gräfin`, `fürst`, `prinz` (#709) ∈ `PARTICLES` ∩ `TITLES`, and `do`/`mc`/`vd` ∈ `PARTICLES` ∩ `SUFFIX_ACRONYMS`; `abd` = "ABD" ∈ `SUFFIX_ACRONYMS`; `abu` ∈ `PARTICLES` ∩ `BOUND_GIVEN_NAMES` (position-dependent: leading token → given-name join, mid-name → family-name join). Usually position-dependent and harmless, but can force a guard or an exclusion (the `last_base` all-particles guard; `abd`, kept out of `BOUND_GIVEN_NAMES` until decisions/P5 found the credential collision cost nothing at either end). ## Step 2: real names, not memory diff --git a/tests/v2/cases.py b/tests/v2/cases.py index c4414ee5c..ab5484f79 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -9129,9 +9129,10 @@ def _check_cjk_shape_purity(self) -> None: ambiguities=(), classification="fix(#613)", notes="decision (4) of #613: the part after the comma is a " - "suffix behind 'Smith Jr.', whose one name word reads " - "positionally as 'Smith Jr.' alone does, with nothing to " - "report. b547c247 read given 'Smith' and reported " + "suffix behind 'Smith Jr.', and the comma settles its one " + "name word as the family, with nothing to report, where " + "'Smith Jr.' alone reads given 'Smith' and reports " + "given-or-family (rules/O5). b547c247 read given 'Smith' and reported " "given-or-family, assign's second count having made the " "comma's positional read of one word"), Case("a_title_chain_after_the_comma_stays_one_title", diff --git a/tests/v2/pipeline/test_vocab.py b/tests/v2/pipeline/test_vocab.py index e363aa670..aa5f98fac 100644 --- a/tests/v2/pipeline/test_vocab.py +++ b/tests/v2/pipeline/test_vocab.py @@ -450,6 +450,7 @@ def test_period_joined_vocab_retires_the_single_character_chunk() -> None: ("Ma.Ed", False), # members alone, nothing settles them ("M.Filho", False), # a suffix word that is also a name (S11) ("Msc.Filho", False), + ("MD.Filho", False), # the example under period_joined_credential's S11 quote; Msc.Filho's path ("X.Y.I.", False), # the single-character exception ("BM.BS", False), # 'bm' unlisted ("PhD", False), # no period, no chunks diff --git a/tests/v2/rules_budget.json b/tests/v2/rules_budget.json index c1398fe8d..92117869e 100644 --- a/tests/v2/rules_budget.json +++ b/tests/v2/rules_budget.json @@ -37,7 +37,7 @@ "O1": {"words": 152, "then_words": 92, "because_sentences": 1, "except": 0, "precedence": 0, "accepted": 1, "accepted_words": 122}, "O2": {"words": 115, "then_words": 60, "because_sentences": 1, "except": 0, "precedence": 0, "accepted": 2, "accepted_words": 66}, "O3": {"words": 47, "then_words": 18, "because_sentences": 1, "except": 0, "precedence": 0, "accepted": 0, "accepted_words": 0}, - "O4": {"words": 91, "then_words": 57, "because_sentences": 1, "except": 0, "precedence": 0, "accepted": 0, "accepted_words": 0}, + "O4": {"words": 102, "then_words": 68, "because_sentences": 1, "except": 0, "precedence": 0, "accepted": 0, "accepted_words": 0}, "O5": {"words": 614, "then_words": 291, "because_sentences": 2, "except": 6, "precedence": 1, "accepted": 1, "accepted_words": 64}, "O6": {"words": 903, "then_words": 242, "because_sentences": 2, "except": 7, "precedence": 4, "accepted": 8, "accepted_words": 321}, "O7": {"words": 710, "then_words": 189, "because_sentences": 2, "except": 5, "precedence": 3, "accepted": 7, "accepted_words": 244},