M5: tier 1 translated end to end

[tamnd]

Tier 1 translated end to end. Spends model calls.

bugs.po, the 17 files of tutorial/ and library/functions.po. Roughly 1 700 entries and 60 batches.

Tier 1 is not first because it is easiest. It is first because the PEP 545 language-switcher condition is exactly that set, and because 1 700 entries is small enough that a person can read all of it and find every systematic problem before the later tiers make the same mistake 80 000 times.

Checklist

• prompts/translate_vi.md, with three worked examples taken from the human translations rather than invented
• render.py, translate.py, and the three-step retry ladder
• apply.py with precedence and --check
• audit/ with every check that applies to a translated corpus
• report.py: coverage and quality
• Content repo audit.yml live: audit, report freshness, apply --check
• Run tier 1 and record the refusal rate per invariant and per attempt

Exit

Tier 1 is in the content repo, every entry fuzzy and carrying a provenance comment, pydocvi audit passes every hard check, and the refusal rates per rule are in a comment on this issue. apply --check is byte-identical in CI.

[tamnd]

#27 is merged. The first two boxes are done: the prompt is a file in the
package, render.py turns a batch into the two messages, and translate.py
reads an answer back into translations, refusals and the next attempt.

Numbers from this branch, all measured rather than assumed.

The design notes are wrong about marker reordering. They say moving a marker
is usually right because Vietnamese does not put modifiers where English does.
In the human translations, of the 222 entries carrying two or more distinct
markers where every marker survived byte-identically, the translators reordered
markers 0 times. Single markers do move, almost entirely inside headings.
The prompt was written to the measurement: a marker may move, and reordering
several in one sentence is a sign to translate more literally instead. P01 still
never checks order.

Two bugs found by running the thing rather than by reading it.

	Entries affected	Batches affected	What it did
#25 numbered line read as a marker	16 of 73 413	12 of 2 776	Silently truncated 7 of the 16
#26 P09 against the unrestored answer	617 of 73 413	391 of 2 776	Refused a perfect translation, three rungs, then dead

#25 was filed saying the stray index kills the batch through P03. Running it
showed worse for some of them: when the stray is an index the answer has already
used it is skipped, the indices stay 1..N, P03 never fires and nothing reports
it. That is 7 of the 16, silently truncated. The other 9 carry an index outside
1..N and were at least reported. All 16 now survive whole.

The counts here are the corrected ones. The commit message and #25 say 29
entries, which is a figure I cannot reproduce and did not check before writing
it down. The measurement is a numbered line at the start of a line inside an
entry body, counted over the batched prose entries, and survival is simulated by
having the answer echo each protected body verbatim under its own index.

#26 was the expensive one. A format specifier is a protected span, so P09 was
comparing a source holding %s against a translation holding the marker that
replaced it. Left in, 391 batches would each have carried at least one entry
that could not pass, costing two extra calls apiece before dying. At the 166.8
calls an hour measured in #9, that is about 4.7 hours of fleet time spent
producing nothing, and 617 entries left in English.

Refusal rates against the fleet are not in this comment, because nothing
here has called a model yet. They come with the last box.

Next: apply.py with precedence and --check.

[tamnd]

apply is in, PR #29. Numbers below are from a run over the real 3.15 corpus, not from the test fixtures.

The run

548 files, 87,008 entries, against the 968 human segments currently in the memory.

$ pydocvi apply
304 written   1,475 left to the reviewer   85,229 still English
548 file(s) changed                                        5.8s

$ pydocvi apply --check
0 written   1,779 left to the reviewer   85,229 still English
the catalogs are what the memory says they should be       7.7s   exit 0

Clean on the first check after the first write, which is the property the whole module is shaped around. The second run reads 1,779 rather than 1,475 because the 304 it just wrote are human segments, written without a fuzzy flag, and on the way back in they read as exactly what they are.

85,229 still in English is the honest state of things. The memory holds 968 segments and nothing has been translated yet, so this run is only the human corpus landing in the right files. It is the plumbing check, not the translation.

Precedence, measured

The content repo already carries a Transifex snapshot at these paths, so there was real work to run over rather than a fixture pretending there was.

reviewed entries: 1,475 untouched, 0 altered, 25 dropped as no longer upstream

Untouched means byte-identical, not equal after re-rendering. A reviewed entry is returned with the physical lines it was read from, so it contributes zero bytes to the diff and cannot be reflowed by a run that had nothing to say about it.

The 25 are entries the target has and 3.15 does not. Upstream owns which strings exist, so they go. about.po has one of them: the acknowledgements paragraph now points at the GitHub contributors graph instead of Misc/ACKS, which makes it a different string with a different segment id, and its Vietnamese translation is not a translation of the new one.

Two decisions I want on the record

Refusing the entry, not the run. Spec 01 §3 says apply refuses to run rather than overwrite a person's work. I refuse the entry instead. The guarantee is the same, no machine string ever lands on a human one, and this version is one you can actually operate: a reviewer unfuzzying a string is the normal case, and refusing the whole run would leave the pipeline red until the memory caught up with the catalog.

No clock. apply reads no clock and calls no model. Every provenance field comes from the segment, none from the run doing the writing, and --check renders each file with the revision date that file already carries. I had both of those wrong first and found it by diffing rendered output against disk: a check that reports the whole corpus as changed the day after it was written is a check that stops being read.

Known consequence

X-Generator is in the header apply owns, so bumping the tool version makes --check report all 548 files as changed until a re-apply. That is true rather than a bug, but it means a release lands with a re-apply in the same PR.

Tests

33 new, 945 in the suite, apply.py at 100% line and branch.

Next box is audit/.

[tamnd]

Second of the three fixes from the first real run, merged as #31. No checklist box for this one: it is not on the list because nobody expected to need it.

doctor used to say a run could start on the strength of GET /v1/health. server2 answers that with 200 and then never finishes a completion, so doctor was green while the queue did not move. The check now asks every healthy route for one short completion through the same client the worker uses, because whether a job gets an answer is not a question you can ask any other way.

Against the fleet just now:

server3: up 127.0.0.1:8103 -> server3:8077  health 200
server2: up 127.0.0.1:8102 -> server2:8077  health 200
asking 2 route(s) to complete a call, a minute or so
server3: answered in 24s, served by gpt-5-6
server2: no answer, nothing within 120s
server3 was asked for gpt-5 and served gpt-5-6
1 of 2 routes completing calls

Two minutes wall clock, most of it waiting out server2. Exit 0, which is right, since one live route is enough. --quick skips the calls for anyone who only wants to know about tunnels.

The second line of output is the other thing this turned up. The proxy ignores the model field: gpt-5, gpt-5-mini, gpt-4.1 and gpt-4o-mini were all asked for during the route comparison and gpt-5-6 came back for all four. routes.json saying model: gpt-5 is a fiction, and the exit criterion above says every tier 1 entry carries a provenance comment. Writing model=gpt-5 into those comments would be recording something that did not happen. Answer.served now carries what actually answered and answered_by is what provenance will record. That last bit of wiring waits on the translate command, since nothing writes provenance yet.

Where the remaining boxes stand. audit/, report.py and audit.yml are untouched. The run itself is blocked on there being no pydocvi translate command at all: translate.py is a library module, and the route comparison that produced these findings had to drive it from a script outside the repo. That command is next, and the run and its refusal rates follow it.

[tamnd]

Tier 1, run for real

Two runs against server3 and server2, an hour and a half of wall clock, and the thing worth writing down is not the refusal rate. It is what the refusals turned out to be.

The run

Dry run first: tier 1: 32 batches, 746 entries, 32 queued, and about 0.4h at the measured 100.7 calls/hour. 56 tier 1 batches exist; 24 were already wholly in the memory from the human translations, so they were dropped before anything was queued.

78 calls in 1.5h at 50.8 calls/hour, 0 empty, 0 failed
718 entries accepted, 114 refused, 1 batches refused whole, 28 entries out of rungs

rule	count
P01	22
P02	22
P03	1
P05	90
P08	71

rung	refusals
1 first	107
2 named	51
3 alone	48

The memory went from 968 human and 0 machine to 968 human and 658 machine, 1,626 in total.

The estimate was twice as fast as the run

50.8 calls/hour measured against 100.7 from fleet bench. The bench sends a one-word prompt and a real batch is 40 entries, so the bench is measuring the round trip and the run is measuring the work. The number translate prints before it asks for confirmation is therefore about 2x optimistic, and a full corpus pass is closer to 54 hours than to the 31.7 the bench report claims. That report wants a realistic prompt before its number is worth quoting.

78 per cent of the refusals were code that should never have been sent

P05 is "empty, or identical to the source" and P08 is "long string with no Vietnamese diacritic". Together they were 161 of the 206 rule hits.

Every one of the 26 tier 1 entries that ended with nothing in the memory was a block of code that classify() called prose:

x = MyClass()
import sound.effects.echo
source tutorial-env/bin/activate
$ chmod +x myscript.py
__all__ = ["echo", "surround", "reverse"]
squares = [x**2 for x in range(10)]

The model returned them unchanged, which is correct. P05 refused the answer for being identical to the source, P08 refused it for having no Vietnamese in it, and the ladder then spent two more calls on each one getting the same correct answer twice more.

This is the answer to why so much was left untranslated, and it is not the prompt and not the model. The classifier's block rule needs at least one indented line, and a block that never indents is invisible to it.

Fixed in #33. A line that looks like code counts as a literal block and an entry counts when every line of it does, with the shapes measured against all 87,008 entries rather than guessed: 917 entries move out of prose and none move into it. Tier 1 drops from 26 unfinished entries to 2, and from 32 outstanding batches to 2.

Two ways the queue lost the record

Also #33, and also only visible from a real run.

queue stats ended the first run at pending 0, leased 0, done 78, dead 0 while the tally reported 28 entries out of rungs. The worker hands the answer to the stage, the stage buries the job, and then the worker finishes the job because the call did come back, unlinking the dead file. A02 exists so that no dead job goes unaccounted for and would have found nothing.

The second run then made only three calls and reported no entries out of rungs at all. Job ids are content-addressed on the rung and the advice as well as the entries, so the second run rebuilt the identical ladder and add refused every step of it as already known. That is the property that makes a second pass cheap; it also meant nothing climbed and nothing was buried. A rung nothing can be queued for is now a rung that is spent.

What is left in tier 1

Two entries, both filed: #34 for f-string, which wants a keep_en glossary row and a P05 that knows about it, and #35 for a space-padded table of tuple comparisons that every code rule misses for a reason worth keeping.

[tamnd]

audit/ is in, #36. Forty-one checks in six groups, no model calls, no sockets, no key. Every hard check is computable from the catalogs plus the upstream pin plus the glossary, which is what makes it fast enough for every push.

What it finds on the real corpus

Run against the content repo as it stands, not a fixture.

check	findings	what it is
S08	548	header drift, every file
A03	388	the memory holds a machine translation the catalog never received
H04	382	file ends in more than one newline
L07	306	written by gpt-5-6-mini though all three routes ask for gpt-5
S03	228	upstream drift
P07	48	doctest with a translated comment
P04	46	model added a leading space
S02	28	msgid edited here rather than upstream
H03	0	no key-shaped string is tracked

The 306 is the one that matters and it now has its own issue, #37. All three routes request gpt-5 and 306 committed entries came back from something else. It was found by a check written to catch a plumbing bug in answered_by, which incidentally proves the plumbing works: the model name reaching the memory is the one the server reported, not the one we sent.

The 382 in H04 were checked by hand with xxd before believing them. They are genuine trailing blank lines, not a parser artifact.

L04, a hard check that could not fire

Worth writing down because it is the kind of thing that only turns up when you test a check rather than run it.

L04 was "no English sentence copied through". It re-derived the classifier's verdict from the msgid and looked for English function words in whatever came back. A no-op is defined as a string with no run of two letters left once the markup is stripped, so a function word cannot survive in one, and the check was logically incapable of producing a finding. Measured over the real content repo: zero hits across all 77 839 distinct msgids, by construction rather than by the corpus being clean. It had been sitting in the hard set reporting a pass.

Re-scoping it to literal blocks was measured and rejected: 162 candidates, every one a license text (PSF, BeOpen, zlib, W3C, WIDE, MT19937), all deliberately untranslated. A hard check that fails on correct content is a check somebody switches off.

It now compares what the corpus recorded in its passthrough= provenance comment against what the classifier says today, which is the only signal that actually drifts. Only prose is reported. A string that was a no-op and is now a literal block still reads correctly; a string that is now prose is one the corpus is showing a Vietnamese reader in English.

The tone-mark fix

Several language checks were matching Vietnamese with [a-z], which in re on str is ASCII only, so every word with a tone mark fell out of the match. Vietnamese tone marks are letters, not decoration: bạn is "you" and bản is "copy", anh is "elder brother" and ảnh is "image". Every check that touches Vietnamese now works on NFC-normalized, casefolded, tone-marks-intact text and matches with [^\W\d_]+. Before the fix L05 reported almost nothing, because the pronouns it looks for all carry tone marks.

Tests

174 new tests, one file per group plus the model, and nine for the CLI. Local gate: 1 273 passed, 97.17% coverage against an 85% floor, make secrets clean.

Two of the CLI tests were passing for the wrong reason and are fixed here. They wrote a .po file with no header entry, which the parser refuses before any check gets to look, so the exit code came from the parse failure rather than from S02. They now apply a real catalog and edit one msgid.

Next: report.py, then the content repo's audit.yml.