Skip to content

Make an identifier need a dot, an underscore or a digit - #47

Merged
tamnd merged 1 commit into
mainfrom
narrow-the-identifier-rule
Aug 18, 2026
Merged

Make an identifier need a dot, an underscore or a digit#47
tamnd merged 1 commit into
mainfrom
narrow-the-identifier-rule

Conversation

@tamnd

@tamnd tamnd commented Aug 18, 2026

Copy link
Copy Markdown
Owner

is_version_marker matched any single token of identifier characters, and a single English word is a single token of identifier characters. So Footnotes was an identifier. So were Availability, Examples, Meaning, Exceptions, Author and 1 335 other ordinary words: 9 366 entries over the corpus, every one of them a heading or a table cell that wants translating.

What it cost

The first paragraph of classify.py says a false positive here leaves an English sentence sitting in the corpus wearing a translation's clothes. That is not a hypothetical any more, and this rule is what produced it. 2 808 of those entries were copied through from the msgid and stamped passthrough=version_marker, which is a claim that the string needs no translation.

Written back as English Entries
Availability 62
Exceptions 25
Author 21
Introduction 18
Description 15
Notes 11

English headings written into a Vietnamese catalog and certified by the tool that wrote them.

The other 6 558 already had a person's translation, and those were hidden a different way. A non-translatable kind is excluded from Corpus.prose(), so no check that reads a translation was looking at any of them. 41 of the words are rendered more than one way across 988 entries, and the disagreements are not stylistic.

Word Renderings
os hệ điều hành in all 28, which is the operating system rather than the module
sys hệ thống in 38, sys in 20
object sự vật in 50 and vật thể in 23, both a physical thing rather than the computing sense
string chuỗi in 34, sợi dây in 8, which is a length of rope
statement tuyên bố in 19, which is a public declaration, câu lệnh in 9
file tập tin in 18, tệp tin in 12, tài liệu in 6

Why narrowing it is safe

The protection that first paragraph describes was never this function's doing. :mod:`asyncio` and ``sys`` reach is_noop, because stripping the markup leaves nothing behind. This function only ever sees a word with no markup on it, and for those the safe direction is the one the module already states: a wasted call costs a call, and an English heading in a Vietnamese page is not noticed until a reader meets it. There is a test for that now, so the reasoning is checked rather than asserted in a comment.

The rule

A dot, an underscore or a digit somewhere in it. That keeps os.path, __init__, size_t, PyMem_RawMalloc and 3.14, and lets go of every bare word. The dotted form is written segment by segment rather than as one character class so that a trailing dot does not count, because Success. is a one-word sentence and the old pattern took it.

What it costs

Not free. The 2 808 go back to being untranslated, which is what they are, and a full run grows by 43 batches. In the content repo this turns 2 826 passthrough entries back into empty ones and apply --check reports 311 files, so the corpus needs re-applying and those headings need a real translation before that gate is green again.

Corpus counts re-measured: prose 72 496 to 75 660, version_marker 3 843 to 650, noop 5 692 to 5 721. The 29 that went to noop are single letters, which no rule but the word rule ever explained.

Refs #11

is_version_marker matched any single token of identifier characters, and a
single English word is a single token of identifier characters. So Footnotes
was an identifier. So were Availability, Examples, Meaning, Exceptions, Author
and 1335 other ordinary words: 9366 entries over the corpus, every one of them
a heading or a table cell that wants translating.

The first paragraph of classify.py says a false positive here leaves an English
sentence sitting in the corpus wearing a translation's clothes. That is not a
hypothetical any more and this rule is what produced it. 2808 of those entries
were copied through from the msgid and stamped passthrough=version_marker,
which is a claim that the string needs no translation: Availability 62 times,
Exceptions 25, Author 21, Introduction 18, Description 15, Notes 11. English
headings written into a Vietnamese catalog and certified by the tool that wrote
them.

The other 6558 already had a person's translation and were hidden a different
way. A non-translatable kind is excluded from Corpus.prose(), so no check that
reads a translation was looking at any of them. 41 of the words are rendered
more than one way across 988 entries, and the disagreements are not stylistic:
sys is he thong in 38 and sys in 20, os is he dieu hanh in all 28, object is
su vat in 50 and vat the in 23 where both mean a physical thing, string is
soi day in 8 which is a length of rope, statement is tuyen bo in 19 which is a
public declaration.

The protection classify.py's first paragraph describes was never this
function's doing. :mod:`asyncio` and ``sys`` reach is_noop, because stripping
the markup leaves nothing behind, so narrowing the rule here does not weaken
it. This function only ever sees a word with no markup on it, and for those the
safe direction is the one the module already states.

An identifier now needs a dot, an underscore or a digit in it, which keeps
os.path, __init__, size_t, PyMem_RawMalloc and 3.14. The dotted form is written
segment by segment rather than as one character class so that a trailing dot
does not count, because Success. is a one-word sentence.

Not free: the 2808 go back to being untranslated, which is what they are, and a
full run grows by 43 batches.
@tamnd
tamnd merged commit af855a3 into main Aug 18, 2026
6 of 7 checks passed
@tamnd
tamnd deleted the narrow-the-identifier-rule branch August 18, 2026 04:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant