numfmt: find the suffix by byte, not by character - #14296
Open
arbelonson-source wants to merge 1 commit into
Open
numfmt: find the suffix by byte, not by character#14296arbelonson-source wants to merge 1 commit into
arbelonson-source wants to merge 1 commit into
Conversation
`find_valid_number_with_suffix` ends the number at a byte offset and then
looks for the suffix that many *characters* further in. The two agree
only while every character is one byte, which the decimal separator is
not in a locale like ar-SA, where it is the two-byte U+066B:
$ LC_ALL=ar_SA.UTF-8 numfmt --from=si '1٫€K'
thread 'main' panicked at src/uu/numfmt/src/format.rs:70:20:
byte index 4 is not a char boundary; it is inside '€' (bytes 3..6)
The number is `1٫`, three bytes. Skipping three characters walks past
the `€` and finds the `K`, so `K` is taken as the suffix; slicing to
three bytes plus one then cuts the `€` in half and the slice aborts the
process. GNU rejects the input and exits 2.
Take the suffix from the bytes after the number instead, so the two
accountings cannot disagree. The two arms that returned the same slice
are now one, since duplicating the arithmetic is what let it drift.
Fixes uutils#13937.
|
GNU testsuite comparison: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #13937.
find_valid_number_with_suffixends the number at a byte offset and then looks for the suffix that many characters further in. The two agree only while every character is one byte — which the decimal separator is not in a locale like ar-SA, where it is U+066B٫, two bytes:The number here is
1٫, three bytes but two characters. Skipping three characters walks past the€and lands on theK, soKis accepted as the suffix; slicing to three bytes plus one then cuts the€in half and thestrslice aborts the process. GNU rejects the input and exits 2.The fix is to take the suffix from
s[numeric_part.len()..], so there is only one accounting. The two arms that returned the same slice are folded into one, since duplicating the index arithmetic three times is what let it drift.@leeewee's write-up in the issue has the full table of the three arms and the input shape that reaches each — this covers all three.
Behaviour
Every affected input now exits 2 as GNU does, instead of aborting:
LC_ALL=ar_SA.UTF-8)numfmt --from=si '1٫€K'numfmt: invalid suffix in input '1٫€K': '€K'numfmt --from=si '1٫€Kx'numfmt: invalid suffix in input '1٫€Kx': '€Kx'numfmt --from=auto '1٫€Ki'numfmt: invalid suffix in input '1٫€Ki': '€Ki'numfmt --from=si '1٫Ki'... '1٫Ki': 'Ki'... '1٫Ki': 'i'The last row is the same desync seen from the other side:
Kis the suffix andiis what is wrong with the input, but the character-skip had landed early and blamed both.That
numfmtaccepts1٫5Kat all where GNU rejects it is a separate matter, tracked in #14232; this PR does not change it.Testing
--fromin all five modes and--toin four, crossed with 40 inputs (plain, suffixed,i-suffixed, malformed, and the multibyte shapes) underLC_ALLofC,ar_SA.UTF-8,de_DE.UTF-8andfr_FR.UTF-8, plus--padding,--suffixand--field: 860 identical, 16 changed, and all 16 are underar_SA.UTF-8— the 12 aborts above, plus the 41٫Kicases. Nothing outside the multibyte-separator locale moves.ar_SA.UTF-8: every changed case now agrees with GNU on the exit code.test_locale_multibyte_separator_before_a_multibyte_char, verified to fail onmain.cargo test --features numfmt --test tests -- test_numfmt: 186 passed, 0 failed (185 pre-existing, 1 new);cargo test -p uu_numfmt: 53 passed.cargo fmt --checkandcargo clippy -p uu_numfmt --all-targets -- -D warnings: clean.Disclosure
Prepared with AI assistance (Claude Opus 5, via Claude Code), per the AI policy in CONTRIBUTING.md. GNU's behaviour was established by running the installed GNU binary as a black box; I did not read GNU coreutils source. All testing was run locally.