Found while shipping M4 (#10). The deterministic checks say a row is well formed. They cannot say it is right. file rendered "tệp tin" is well formed, passes every rule, and is a word the human translators never once wrote.
The only ground truth available is work humans already did, being the 1,435 human-translated entries in the content repo. I measured the finished contract against them by hand, with a throwaway script, and it scored 975 of 1,247 occurrences, 78%. That measurement found five wrong rows that nothing else was ever going to find:
| term |
the row said |
humans write |
score |
| file |
tệp tin |
tệp |
12 of 56 |
| iterable |
có thể lặp |
khả lặp |
0 of 25 |
| slice |
lát cắt |
phần cắt |
0 of 6 |
| index |
chỉ mục |
chỉ số |
0 of 5 |
| docstring |
chuỗi tài liệu |
docstring |
0 of 4 |
It also settled two places where the curated rows and the hand-written style rules in GLOSSARY.md disagreed, by counting which one the humans actually followed.
A number I measured once with a script in /tmp is not a number anyone can trust. It should be a command.
What to build
pydocvi glossary agree, deterministic, no model calls. For each row, find the human-translated entries whose msgid contains the English term using the existing matcher, then count how many of those entries' msgstr contain the Vietnamese rendering. Report per row and in total.
It shares the matcher with the prompt builder, G02 and stale --glossary, which is the standing rule for that code and the reason this is cheap to write.
What it must not do
Not fail the build on a low score. Agreement is evidence, not a verdict. A row can be right and score zero because the human paraphrased, or because the rendering needs "các" in front, which is exactly what happened to the three rows that looked worst in my run. The command sorts by how bad the disagreement looks and leaves the judgement to a person.
Not rewrite rows. It reads the content repo and the term list and writes nothing.
Threshold worth reporting
Rows with zero agreement across three or more occurrences are the interesting ones. Everything below three occurrences is noise. Sort those to the top and print the human's actual msgstr beside the row so the reader can see what was written instead.
Found while shipping M4 (#10). The deterministic checks say a row is well formed. They cannot say it is right.
filerendered "tệp tin" is well formed, passes every rule, and is a word the human translators never once wrote.The only ground truth available is work humans already did, being the 1,435 human-translated entries in the content repo. I measured the finished contract against them by hand, with a throwaway script, and it scored 975 of 1,247 occurrences, 78%. That measurement found five wrong rows that nothing else was ever going to find:
It also settled two places where the curated rows and the hand-written style rules in
GLOSSARY.mddisagreed, by counting which one the humans actually followed.A number I measured once with a script in
/tmpis not a number anyone can trust. It should be a command.What to build
pydocvi glossary agree, deterministic, no model calls. For each row, find the human-translated entries whosemsgidcontains the English term using the existing matcher, then count how many of those entries'msgstrcontain the Vietnamese rendering. Report per row and in total.It shares the matcher with the prompt builder,
G02andstale --glossary, which is the standing rule for that code and the reason this is cheap to write.What it must not do
Not fail the build on a low score. Agreement is evidence, not a verdict. A row can be right and score zero because the human paraphrased, or because the rendering needs "các" in front, which is exactly what happened to the three rows that looked worst in my run. The command sorts by how bad the disagreement looks and leaves the judgement to a person.
Not rewrite rows. It reads the content repo and the term list and writes nothing.
Threshold worth reporting
Rows with zero agreement across three or more occurrences are the interesting ones. Everything below three occurrences is noise. Sort those to the top and print the human's actual
msgstrbeside the row so the reader can see what was written instead.