Skip to content

sort: apply -n to the key, not the whole line - #14331

Open
Socialpranker wants to merge 1 commit into
uutils:mainfrom
Socialpranker:sort-numeric-whole-line-fast-path
Open

sort: apply -n to the key, not the whole line#14331
Socialpranker wants to merge 1 commit into
uutils:mainfrom
Socialpranker:sort-numeric-whole-line-fast-path

Conversation

@Socialpranker

Copy link
Copy Markdown
Contributor
$ printf '19\n21\n' | sort -n -k1.2     # GNU
21
19
$ printf '19\n21\n' | sort -n -k1.2     # uutils, before
19
21

-k1.2 selects from the second character on, so the keys are 9 and 1 and
21 sorts first. uutils compared 19 against 21 instead.

-n parses a number out of every line once, up front, and compare_by compares
those numbers before it reaches the key loop. That is the same comparison only
when the key happens to span the whole line, and nothing checked that it did.
--debug shows the key was selected correctly all along -- it was simply not
what got compared:

$ printf '19\n21\n' | uutils sort --debug -n -k1.2
19
 _        <- the key is "9"
21
 _        <- the key is "1"
        ...and the output is still 19, 21

Two more shapes of the same defect:

$ printf '1.9\n2.1\n' | sort -n -t. -k2   # keys "9" and "1"
GNU: 2.1, 1.9        uutils: 1.9, 2.1

$ printf '19\n21\n3\n' | sort -n -k1r     # the key reverses on its own
GNU: 3, 21, 19       uutils: 3, 19, 21

The last one cannot work through the shortcut at all: it compares with the
global ordering, and a key's own r never reaches it.

It also broke the last-resort comparison. When every key is equal, GNU falls
back to comparing the lines byte by byte:

$ printf '1\n10\n2\n20\n3\n' | sort -n -k1.2   # every key empty or "0"
GNU: 1, 10, 2, 20, 3       uutils: 1, 2, 3, 10, 20

The fix

can_use_whole_line_numeric guards the shortcut, in the shape the neighboring
can_use_fast_lexicographic and can_use_fast_ascii_insensitive already use:
one key, numeric, not reversed, starting at the first character of the first
field and running to the end of the line. Plain -n and -n -k1 still take it,
so the optimization is kept where it is correct -- sort -n over 300k shuffled
integers is unchanged (0.051s patched vs 0.079s before, same ballpark, the
shortcut is still in play). sort -n -k1.2 goes the long way now and costs
about 18% more, which is the price of the right answer.

-h, -g, -M and -V never had a whole-line shortcut, which is why they
were already correct here.

How the GNU behavior was established

By running the installed GNU binary as a black box and recording its output --
every transcript above is from debian:stable-slim with GNU coreutils 9.7. I
did not read GNU coreutils source.

Testing

  • cargo test --features sort --no-default-features -- test_sort: 188 passed,
    0 failed
    (187 pre-existing, 1 new). No existing test needed a change.
  • New test_numeric_sort_uses_the_key_not_the_whole_line covers the character
    offset, the field, the key-local r, the last-resort fallback, and the two
    whole-line cases that must keep working.
  • Mutation check: reverting sort.rs and chunks.rs alone (test file
    untouched) makes the new test fail.
  • cargo fmt --all --check and cargo clippy -p uu_sort --all-targets: clean.
  • Differential A/B against GNU coreutils 9.7: 12 inputs (integers, floats,
    leading blanks, negatives, nan/inf, comma- and dot-separated fields,
    non-numeric lines) x 21 flag combinations: 33 cases moved from differing to
    byte-identical, 0 regressions
    .

Two cases still differ, identically before and after this change, so they are
pre-existing and unrelated: sort -n -c truncates the offending line in its
diagnostic (1.10 reported as 1.1, nan as NaN).

Also noticed while sweeping, and left alone as a separate question: GNU treats a
key's b as an ordering option, so sort -n -k1b sorts lexicographically,
while uutils applies the global -n to it. That is about which modifiers count
as ordering options, not about which text gets compared.

Disclosure

Prepared with AI assistance (Claude Opus 5, via Claude Code), per the AI policy
in CONTRIBUTING.md. Every GNU behavior quoted above came from running the
installed binary, not from reading GPL source. All testing was run locally.

`-n` parses a number from every line up front and compares those numbers
before any key is looked at. That shortcut is only the same comparison
when the key spans the whole line, but nothing checked that:

    printf "19\\n21\\n" | sort -n -k1.2

sorts on "9" and "1", so GNU prints 21 first; uutils compared 19 against
21 and printed 19. The same went for a key picked out by field
(`-n -t. -k2`), and for a key with its own `r`, which the shortcut could
not honor because it only knows the global ordering.

Guard the shortcut with `can_use_whole_line_numeric`, in the shape the
neighboring `can_use_fast_lexicographic` already uses: a single key,
numeric, not reversed, starting at the first character of the first
field and running to the end of the line. Plain `-n` and `-n -k1` still
take it.
@github-actions

Copy link
Copy Markdown

GNU testsuite comparison:

Skipping an intermittent issue tests/cut/bounded-memory (passes in this run but fails in the 'main' branch)
Skipping an intermittent issue tests/date/date-locale-hour (passes in this run but fails in the 'main' branch)
Skipping an intermittent issue tests/tail/follow-name (passes in this run but fails in the 'main' branch)
Skip an intermittent issue tests/pr/bounded-memory (was skipped on 'main', now failing)

@sylvestre

Copy link
Copy Markdown
Contributor

Differential A/B against GNU coreutils 9.7

it is old, please try with master binaries
(or at least 9.11)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants