Make an identifier need a dot, an underscore or a digit - #47
Merged
Conversation
is_version_marker matched any single token of identifier characters, and a single English word is a single token of identifier characters. So Footnotes was an identifier. So were Availability, Examples, Meaning, Exceptions, Author and 1335 other ordinary words: 9366 entries over the corpus, every one of them a heading or a table cell that wants translating. The first paragraph of classify.py says a false positive here leaves an English sentence sitting in the corpus wearing a translation's clothes. That is not a hypothetical any more and this rule is what produced it. 2808 of those entries were copied through from the msgid and stamped passthrough=version_marker, which is a claim that the string needs no translation: Availability 62 times, Exceptions 25, Author 21, Introduction 18, Description 15, Notes 11. English headings written into a Vietnamese catalog and certified by the tool that wrote them. The other 6558 already had a person's translation and were hidden a different way. A non-translatable kind is excluded from Corpus.prose(), so no check that reads a translation was looking at any of them. 41 of the words are rendered more than one way across 988 entries, and the disagreements are not stylistic: sys is he thong in 38 and sys in 20, os is he dieu hanh in all 28, object is su vat in 50 and vat the in 23 where both mean a physical thing, string is soi day in 8 which is a length of rope, statement is tuyen bo in 19 which is a public declaration. The protection classify.py's first paragraph describes was never this function's doing. :mod:`asyncio` and ``sys`` reach is_noop, because stripping the markup leaves nothing behind, so narrowing the rule here does not weaken it. This function only ever sees a word with no markup on it, and for those the safe direction is the one the module already states. An identifier now needs a dot, an underscore or a digit in it, which keeps os.path, __init__, size_t, PyMem_RawMalloc and 3.14. The dotted form is written segment by segment rather than as one character class so that a trailing dot does not count, because Success. is a one-word sentence. Not free: the 2808 go back to being untranslated, which is what they are, and a full run grows by 43 batches.
This was referenced Aug 18, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
is_version_markermatched any single token of identifier characters, and a single English word is a single token of identifier characters. SoFootnoteswas an identifier. So wereAvailability,Examples,Meaning,Exceptions,Authorand 1 335 other ordinary words: 9 366 entries over the corpus, every one of them a heading or a table cell that wants translating.What it cost
The first paragraph of
classify.pysays a false positive here leaves an English sentence sitting in the corpus wearing a translation's clothes. That is not a hypothetical any more, and this rule is what produced it. 2 808 of those entries were copied through from themsgidand stampedpassthrough=version_marker, which is a claim that the string needs no translation.AvailabilityExceptionsAuthorIntroductionDescriptionNotesEnglish headings written into a Vietnamese catalog and certified by the tool that wrote them.
The other 6 558 already had a person's translation, and those were hidden a different way. A non-translatable kind is excluded from
Corpus.prose(), so no check that reads a translation was looking at any of them. 41 of the words are rendered more than one way across 988 entries, and the disagreements are not stylistic.oshệ điều hànhin all 28, which is the operating system rather than the modulesyshệ thốngin 38,sysin 20objectsự vậtin 50 andvật thểin 23, both a physical thing rather than the computing sensestringchuỗiin 34,sợi dâyin 8, which is a length of ropestatementtuyên bốin 19, which is a public declaration,câu lệnhin 9filetập tinin 18,tệp tinin 12,tài liệuin 6Why narrowing it is safe
The protection that first paragraph describes was never this function's doing.
:mod:`asyncio`and``sys``reachis_noop, because stripping the markup leaves nothing behind. This function only ever sees a word with no markup on it, and for those the safe direction is the one the module already states: a wasted call costs a call, and an English heading in a Vietnamese page is not noticed until a reader meets it. There is a test for that now, so the reasoning is checked rather than asserted in a comment.The rule
A dot, an underscore or a digit somewhere in it. That keeps
os.path,__init__,size_t,PyMem_RawMallocand3.14, and lets go of every bare word. The dotted form is written segment by segment rather than as one character class so that a trailing dot does not count, becauseSuccess.is a one-word sentence and the old pattern took it.What it costs
Not free. The 2 808 go back to being untranslated, which is what they are, and a full run grows by 43 batches. In the content repo this turns 2 826 passthrough entries back into empty ones and
apply --checkreports 311 files, so the corpus needs re-applying and those headings need a real translation before that gate is green again.Corpus counts re-measured: prose 72 496 to 75 660, version_marker 3 843 to 650, noop 5 692 to 5 721. The 29 that went to
noopare single letters, which no rule but the word rule ever explained.Refs #11