uucore: negotiate the requested locale against the translations that exist - #14327
Open
XhstormR wants to merge 1 commit into
Open
uucore: negotiate the requested locale against the translations that exist#14327XhstormR wants to merge 1 commit into
XhstormR wants to merge 1 commit into
Conversation
Contributor
|
@XhstormR sorry but long comment #0 aren't useful. nobody reads such a long comment from a LLM |
sylvestre
reviewed
Aug 31, 2026
…exist
A locale was taken for a translation's file name: whatever LANG named,
stripped of its encoding, was read back as `{locale}.ftl`. The two are
named in different worlds, though. A POSIX environment asks with a
region -- zh_CN, de_DE, es_MX -- while a translation is filed under
whatever distinction its translators needed: a language (de.ftl), a
region (pt-BR.ftl) or a script (zh-Hans.ftl). Only the translations
whose name happened to coincide with a locale's were reachable, so of
53 system locales tried against the 31 translations of ls, 8 read
anything but English. Some two dozen languages were translated and
unreachable, Chinese among them.
Match the two by negotiating, as RFC 4647 and UTS uutils#35 specify and
fluent-langneg implements: enumerate the translations a locales
directory holds, negotiate the request against them, and let the name
of the winner be what everything downstream reads. All 53 then reach
the translation meant for them, and a language added later needs no
code. Turn on fluent-langneg's `cldr` feature for its likely-subtags
data, without which zh_TW, zh_HK and zh_MO read Simplified Chinese --
worse than reading English. It costs 357 KiB of tables, 86% of that
the language-to-script one.
Read the environment the way gettext does while here, since a request
that is never read is a request that cannot be negotiated: LC_ALL and
LC_MESSAGES outrank LANG, LANGUAGE lists locales to try in turn unless
the locale asked for was C or POSIX, and a modifier -- the @euro of
de_DE@euro -- says as little about which language to speak as an
encoding does. Three tests named a language through a variable that
something outranked, and now say so through the one that decides.
None of this was caught because fr-FR is the only translation in this
tree, and `LANG=fr_FR.UTF-8` normalizes to exactly fr-FR: the one
locale whose name coincides with its translation's, and the one every
existing assertion uses.
The embedded translations a WASI build carries are still looked up by
name, as the embedded set cannot be enumerated to negotiate against.
Fixes uutils#12305
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
XhstormR
force-pushed
the
locale-negotiation
branch
from
August 31, 2026 07:42
33916a2 to
fdfe21b
Compare
Author
|
@sylvestre I have simplified the PR description, including only the necessary information. |
|
GNU testsuite comparison: |
Unable to generate the performance reportThere was an internal error while processing the run's data. We're working on fixing the issue. Feel free to contact us on Discord or at support@codspeed.io if the issue persists. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #12305.
LANG=zh_CN.UTF-8never findszh-Hans.ftlbecause the requested locale is used directly as a file name. Requests are region-based (zh_CN,de_DE,es_MX), translations are filed by language or script (de,zh-Hans), so only the names that happen to coincide match. Against the 31 localeslshas in coreutils-l10n, 8 of 53 system locales currently get a translation, 53 with this patch.de_DE,ja_JP,ru_RU,es_MX,en_GBand ~25 others fall back to English too, so this isn't Chinese-specific.Renaming to
zh_CN.ftlas the issue suggests wouldn't work anyway:unic-langidprints-, so the lookup key iszh-CN.ftl.So the patch negotiates instead of matching literally: enumerate the
.ftlfiles in the locales dir, run RFC 4647 filtering over them viafluent-langneg, then read the winner. It also readsLC_ALL/LC_MESSAGES/LANGUAGE, which were ignored before, and strips@modifier.fluent-langnegis already inCargo.lockthroughfluent-bundle. Itscldrfeature pulls in CLDR's likely-subtags tables: +357 KiB on the multicall binary. Without themzh_TW/zh_HK/zh_MOresolve to Simplified, which is worse than English. If that size isn't acceptable, I can generate only the rows needed in build.rs instead (~1 KB) — let me know which you'd rather have.CI never caught this because
fr-FRis the only in-tree translation andfr_FR.UTF-8normalizes to exactlyfr-FR.Three tests picked their language through a variable that
LC_ALLnow outranks, so they setLC_ALLinstead.