Skip to content

shuf: match GNU's RNG and input handling - #198

Open
metif12 wants to merge 1 commit into
vlang:mainfrom
metif12:fix/shuf-main
Open

metif12 wants to merge 1 commit into
vlang:mainfrom
metif12:fix/shuf-main

Conversation

@metif12

@metif12 metif12 commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

Fixes #189 and #193.

This branch contains only the shuf change, so the diff is limited to src/shuf/.
It is the third of a series and should be merged after #195 and #196:

With all three applied the branch is green end to end: v run build.vsh builds
all 76 utilities, make testfmt passes and v test . is 47/47, up from
37/47 on main.

--random-source produced the wrong permutation (#189)

The old code read the random source into memory, summed the bytes, fed that to
a V PRNG, and shuffled with a hand written swap loop. That is not what GNU does,
so the same file gave a different permutation, and the test that checks it
failed.

src/shuf/randint.v is now a direct port of GNU's gl/lib/randint.c and
gl/lib/randperm.c. The leftover entropy in randnum/randmax has to be
carried across calls exactly as GNU does, because that is observable:
shuf -i 1-5 over a one byte source still consumes a single byte, and only
because of that carry.

GNU also switches to a hash based sparse representation for large sparse
permutations. That is purely a memory optimisation — it performs the same
swaps in the same order — so it is not reproduced here.

Verified byte for byte against GNU coreutils 9.4 for:

-i 1-5                -i 0-99 (exhausted)     -n 3 from 5 lines
all lines             -r -n 6 (echo/file/range)  -o
stdin                 -n larger than input    missing newline / missing NUL
empty lines kept      -z

and for the end of file, No such file or directory, invalid input range,
extra operand and cannot combine -e and -i errors.

Other behaviour that was wrong

  • Empty lines were dropped. register_lines_by_file had
    if line.len > 0, so shuf silently lost them.
  • -z did not work. The zero-terminated path appended the whole input as a
    single element instead of splitting on NUL, and open_operands split on
    newline regardless of the flag.
  • -n 0 output every line instead of nothing. The old code used 0 to mean
    "every line", which is exactly what -n 0 is not.
  • -r -n COUNT re-shuffled for every line of output, so it could repeat
    the entire input COUNT times instead of picking COUNT lines.
  • -i accepted anything. split('-') with no validation, so 5-1,
    abc, 1-x and 1-2-3 were all accepted.
  • -e together with -i, and extra operands, were accepted.
  • A missing input file reported V's failed to open file "..." instead of
    GNU's No such file or directory.

Help text (#193)

The option descriptions now use GNU's wording, and the usage lines match GNU's:

Usage: shuf [OPTION]... [FILE]
   or: shuf -e [OPTION]... [ARG]...
   or: shuf -i LO-HI [OPTION]...

The options use the _opt flag variants so V stops appending a
(default false) note to each one, and the value names are LO-HI, COUNT
and FILE as in GNU:

  -e, --echo                treat each ARG as an input line
  -i, --input-range LO-HI   treat each number LO through HI as an input line
  -n, --head-count COUNT    output at most COUNT lines
  -o, --output FILE         write result to FILE instead of standard output
  --random-source FILE      get random bytes from FILE
  -r, --repeat              output lines can be repeated
  -z, --zero-terminated     line delimiter is NUL, not newline

The surrounding boilerplate is still whatever common.flag_parser produces,
because that is what every utility in this repository uses and the README asks
for consistency. Reproducing GNU's exact --help layout would mean bypassing it
and diverging from the other 75 tools. The descriptions, which is what the
issue was about, now match.

Tests

test_random_source no longer hardcodes a permutation. The expected value
1 4 5 2 3 came from GNU 8.32 and does not match any current GNU — 9.4 gives
5 2 1 3 4 for the same input, and I checked that the port reproduces 9.4
exactly. Since the permutation for a given random source is fully determined,
the tests compare against the platform shuf through the rig, which is what
the other tests in this file already do and keeps them valid across GNU
versions.

Added coverage for lines, -n, repeat, exhaustion, a missing source, empty
lines, -z, -o, -n 0, and the input-range and operand errors.

One of the new tests caught a bug while I was writing it: open_operands split
on newline unconditionally, so -z was still wrong after the first round of
fixes. Fixed by threading the delimiter through.

Fixes vlang#189 and vlang#193.

The old code read the random source into memory, summed the bytes, fed that
to a V PRNG and shuffled with a hand written swap loop. That is not what GNU
does, so the same file gave a different permutation and the test that checks it
failed.

src/shuf/randint.v is now a direct port of GNU's gl/lib/randint.c and
gl/lib/randperm.c. The leftover entropy in randnum/randmax has to be carried
across calls exactly as GNU does, because that is observable: shuf -i 1-5 over
a one byte source still consumes a single byte, and only because of that
carry. GNU also switches to a hash based sparse representation for large sparse
permutations; that is purely a memory optimisation and performs the same swaps
in the same order, so it is not reproduced here.

Verified byte for byte against GNU coreutils 9.4 for -i 1-5, -i 0-99, -n 3 from
5 lines, all lines, -r -n 6 in echo/file/range mode, -o, stdin, -n larger than
the input, and the end-of-file and missing-file errors.

* Empty lines were dropped. `register_lines_by_file` had `if line.len > 0`,
  so `shuf` silently lost them.
* `-z` did not work. The zero-terminated path appended the whole input as a
  single element instead of splitting on NUL, and `open_operands` split on
  newline regardless of the flag.
* `-n 0` output every line instead of nothing. The old code used 0 to mean
  "every line", which is exactly what -n 0 is not.
* `-r -n COUNT` re-shuffled for every line of output, so it could repeat the
  whole input COUNT times instead of picking COUNT lines.
* `-i` accepted anything: `split('-')` with no validation, so `5-1`, `abc`,
  `1-x` and `1-2-3` were all accepted. These are now rejected the way GNU
  rejects them, with GNU's message.
* `-e` together with `-i`, and extra operands, were accepted. Both are now
  rejected.
* A missing input file reported V's `failed to open file "..."` rather than
  GNU's `No such file or directory`.

The option descriptions now use GNU's wording, and the three usage lines match
GNU's:

```
Usage: shuf [OPTION]... [FILE]
   or: shuf -e [OPTION]... [ARG]...
   or: shuf -i LO-HI [OPTION]...
```

The options use the `_opt` flag variants so V stops appending a
"(default false)" note to each one, and the value names are LO-HI, COUNT, FILE
and FILE as in GNU. The surrounding boilerplate is still the shape
common.flag_parser gives every utility in this repository; reproducing GNU's
exact --help layout would mean bypassing it and diverging from the rest of the
tools, which the README asks for. The descriptions, which is what the issue
was about, now match.

test_random_source no longer hardcodes a permutation. The expectation
`1 4 5 2 3` came from GNU 8.32 and does not match any current GNU: 9.4 produces
`5 2 1 3 4` for the same input. The permutation for a given random source is
fully determined, so the tests now compare against the platform shuf through
the rig, which is what the other tests in this file already do.

Added coverage for lines, -n, repeat, exhaustion, a missing source, empty
lines, -z, -o, -n 0, and the input-range and operand errors.

`v test .` is now 47/47, was 46/47.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

shuf is currently failing it's own tests

1 participant