manifests/memory.json is 896 KB with 1,435 translated segments out of 87,008. Fully translated it is somewhere north of 30 MB, rewritten in whole on every run that touches it.
H02 already objected the day it was committed, and the fix exempted the one path rather than raising a ceiling, because any number there is a number that gets raised in a hurry by whoever is trying to land a translation run. The exemption is a placeholder for this.
What it should become
manifests/memory/<top-level>.json, one shard per top-level directory of the corpus: library.json, tutorial.json, c-api.json, and a root.json for the files that sit at the top. A run that translates library/functions.po then rewrites one shard of a few megabytes instead of all of it.
Segment identity is a hash of (msgctxt, msgid) and deliberately not file-scoped, which is what lets 3.14 and 3.15 share 85,192 entries for free. So the shard a segment lands in cannot be part of its identity. It is a storage detail: load reads every shard into one store and save writes each segment to the shard for the file it was most recently seen in, with a stable rule for a segment seen in two.
That last part is the only real design question here and it is worth settling before writing anything.
Why not now
896 KB is not a problem, and a sharded store for a file this size is complexity bought against nothing. This becomes real when tier 5 is translated, which is 37,588 entries.
Constraints that carry over
- Still sorted within a shard, still one object per line group. That is what makes the store diff in hunks instead of as a blob, and it is the reason committing it is defensible at all.
apply --check has to keep working, which means load over the shards has to produce exactly the store that save wrote.
manifests/memory.jsonis 896 KB with 1,435 translated segments out of 87,008. Fully translated it is somewhere north of 30 MB, rewritten in whole on every run that touches it.H02already objected the day it was committed, and the fix exempted the one path rather than raising a ceiling, because any number there is a number that gets raised in a hurry by whoever is trying to land a translation run. The exemption is a placeholder for this.What it should become
manifests/memory/<top-level>.json, one shard per top-level directory of the corpus:library.json,tutorial.json,c-api.json, and aroot.jsonfor the files that sit at the top. A run that translateslibrary/functions.pothen rewrites one shard of a few megabytes instead of all of it.Segment identity is a hash of
(msgctxt, msgid)and deliberately not file-scoped, which is what lets 3.14 and 3.15 share 85,192 entries for free. So the shard a segment lands in cannot be part of its identity. It is a storage detail:loadreads every shard into one store andsavewrites each segment to the shard for the file it was most recently seen in, with a stable rule for a segment seen in two.That last part is the only real design question here and it is worth settling before writing anything.
Why not now
896 KB is not a problem, and a sharded store for a file this size is complexity bought against nothing. This becomes real when tier 5 is translated, which is 37,588 entries.
Constraints that carry over
apply --checkhas to keep working, which meansloadover the shards has to produce exactly the store thatsavewrote.