Conversation
This was referenced Oct 2, 2026
Fixes vlang#189 and vlang#193. The old code read the random source into memory, summed the bytes, fed that to a V PRNG and shuffled with a hand written swap loop. That is not what GNU does, so the same file gave a different permutation and the test that checks it failed. src/shuf/randint.v is now a direct port of GNU's gl/lib/randint.c and gl/lib/randperm.c. The leftover entropy in randnum/randmax has to be carried across calls exactly as GNU does, because that is observable: shuf -i 1-5 over a one byte source still consumes a single byte, and only because of that carry. GNU also switches to a hash based sparse representation for large sparse permutations; that is purely a memory optimisation and performs the same swaps in the same order, so it is not reproduced here. Verified byte for byte against GNU coreutils 9.4 for -i 1-5, -i 0-99, -n 3 from 5 lines, all lines, -r -n 6 in echo/file/range mode, -o, stdin, -n larger than the input, and the end-of-file and missing-file errors. * Empty lines were dropped. `register_lines_by_file` had `if line.len > 0`, so `shuf` silently lost them. * `-z` did not work. The zero-terminated path appended the whole input as a single element instead of splitting on NUL, and `open_operands` split on newline regardless of the flag. * `-n 0` output every line instead of nothing. The old code used 0 to mean "every line", which is exactly what -n 0 is not. * `-r -n COUNT` re-shuffled for every line of output, so it could repeat the whole input COUNT times instead of picking COUNT lines. * `-i` accepted anything: `split('-')` with no validation, so `5-1`, `abc`, `1-x` and `1-2-3` were all accepted. These are now rejected the way GNU rejects them, with GNU's message. * `-e` together with `-i`, and extra operands, were accepted. Both are now rejected. * A missing input file reported V's `failed to open file "..."` rather than GNU's `No such file or directory`. The option descriptions now use GNU's wording, and the three usage lines match GNU's: ``` Usage: shuf [OPTION]... [FILE] or: shuf -e [OPTION]... [ARG]... or: shuf -i LO-HI [OPTION]... ``` The options use the `_opt` flag variants so V stops appending a "(default false)" note to each one, and the value names are LO-HI, COUNT, FILE and FILE as in GNU. The surrounding boilerplate is still the shape common.flag_parser gives every utility in this repository; reproducing GNU's exact --help layout would mean bypassing it and diverging from the rest of the tools, which the README asks for. The descriptions, which is what the issue was about, now match. test_random_source no longer hardcodes a permutation. The expectation `1 4 5 2 3` came from GNU 8.32 and does not match any current GNU: 9.4 produces `5 2 1 3 4` for the same input. The permutation for a given random source is fully determined, so the tests now compare against the platform shuf through the rig, which is what the other tests in this file already do. Added coverage for lines, -n, repeat, exhaustion, a missing source, empty lines, -z, -o, -n 0, and the input-range and operand errors. `v test .` is now 47/47, was 46/47.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #189 and #193.
This branch contains only the shuf change, so the diff is limited to
src/shuf/.It is the third of a series and should be merged after #195 and #196:
make testfmtgreen againWith all three applied the branch is green end to end:
v run build.vshbuildsall 76 utilities,
make testfmtpasses andv test .is 47/47, up from37/47 on
main.--random-sourceproduced the wrong permutation (#189)The old code read the random source into memory, summed the bytes, fed that to
a V PRNG, and shuffled with a hand written swap loop. That is not what GNU does,
so the same file gave a different permutation, and the test that checks it
failed.
src/shuf/randint.vis now a direct port of GNU'sgl/lib/randint.candgl/lib/randperm.c. The leftover entropy inrandnum/randmaxhas to becarried across calls exactly as GNU does, because that is observable:
shuf -i 1-5over a one byte source still consumes a single byte, and onlybecause of that carry.
GNU also switches to a hash based sparse representation for large sparse
permutations. That is purely a memory optimisation — it performs the same
swaps in the same order — so it is not reproduced here.
Verified byte for byte against GNU coreutils 9.4 for:
and for the
end of file,No such file or directory,invalid input range,extra operandandcannot combine -e and -ierrors.Other behaviour that was wrong
register_lines_by_filehadif line.len > 0, soshufsilently lost them.-zdid not work. The zero-terminated path appended the whole input as asingle element instead of splitting on NUL, and
open_operandssplit onnewline regardless of the flag.
-n 0output every line instead of nothing. The old code used 0 to mean"every line", which is exactly what
-n 0is not.-r -n COUNTre-shuffled for every line of output, so it could repeatthe entire input COUNT times instead of picking COUNT lines.
-iaccepted anything.split('-')with no validation, so5-1,abc,1-xand1-2-3were all accepted.-etogether with-i, and extra operands, were accepted.failed to open file "..."instead ofGNU's
No such file or directory.Help text (#193)
The option descriptions now use GNU's wording, and the usage lines match GNU's:
The options use the
_optflag variants so V stops appending a(default false)note to each one, and the value names areLO-HI,COUNTand
FILEas in GNU:The surrounding boilerplate is still whatever
common.flag_parserproduces,because that is what every utility in this repository uses and the README asks
for consistency. Reproducing GNU's exact
--helplayout would mean bypassing itand diverging from the other 75 tools. The descriptions, which is what the
issue was about, now match.
Tests
test_random_sourceno longer hardcodes a permutation. The expected value1 4 5 2 3came from GNU 8.32 and does not match any current GNU — 9.4 gives5 2 1 3 4for the same input, and I checked that the port reproduces 9.4exactly. Since the permutation for a given random source is fully determined,
the tests compare against the platform
shufthrough the rig, which is whatthe other tests in this file already do and keeps them valid across GNU
versions.
Added coverage for lines,
-n, repeat, exhaustion, a missing source, emptylines,
-z,-o,-n 0, and the input-range and operand errors.One of the new tests caught a bug while I was writing it:
open_operandsspliton newline unconditionally, so
-zwas still wrong after the first round offixes. Fixed by threading the delimiter through.