Skip to content

io: core/io conformance wave 22 — pipe encodings, write transcoding, readpartial, copy_stream, write_nonblock (+77) - #605

Merged
tannevaled merged 6 commits into
mainfrom
wave22/io
Sep 9, 2026
Merged

tannevaled merged 6 commits into
mainfrom
wave22/io

Conversation

@tannevaled

Copy link
Copy Markdown
Contributor

Advances MRI-Ruby-4.0.x byte-for-byte conformance of the pure-Go (CGO=0) IO
subsystem. Every change was read against MRI io.c (v3_4_0) and the matching
ruby/spec file, and checked byte-for-byte against ruby 4.0.5. Edits are
confined to internal/vm/{io,io_class,io_descriptors,spawn}.go plus new
whitebox _test.go.

core/io ruby/spec: 1149 → 1226 passing (+77) (fail 161→113, error 119→90)

Methods fixed (with reference read)

  • IO.pipe — the read end takes its external/internal encoding from the call's
    encoding arguments (Encoding, name, "ext:int", BOM|, or #to_str), else the
    Encoding defaults captured at creation; the write end carries none; the receiver
    class is honoured so a subclass's .pipe yields subclass instances.
    (rb_io_s_pipe, rb_io_ext_int_to_encs; pipe_spec.rb)
  • IO#write / IO#write_nonblock — an all-empty write returns 0 before any
    closed/writable check; a non-BINARY external encoding transcodes each argument
    (raising UndefinedConversionError/InvalidByteSequenceError) and returns the
    transcoded byte count; the argument String is never mutated. write_nonblock
    is newly defined, modelling a finite pipe buffer so a full pipe reports
    would-block (:wait_writable / Errno::EAGAIN). (io_write, do_writeconv,
    make_writeconv, io_write_nonblock; write_spec.rb + shared/write.rb,
    write_nonblock_spec.rb)
  • Encoding options — a BOM| marker is stripped on :encoding (and mode
    strings) but rejected on :external_encoding / :internal_encoding, matching
    MRI. (rb_io_extract_encoding_option; write_spec.rb)
  • gets / readline / readlines / each_line / IO#sysseek — an oversized-integer
    limit/offset raises RangeError (not TypeError) via the C off_t coercion;
    sysseek raises IOError on a closed stream. (rb_io_getline_1, NUM2OFFT;
    gets_spec.rb, sysseek_spec.rb, shared/pos.rb, shared/readlines.rb)
  • IO.foreach — each yield updates $.; $_ is cleared at EOF; with no block
    it returns a nil-size Enumerator instead of an eager array.
    (rb_io_s_foreach; foreach_spec.rb)
  • IO#readpartial — full io_getpartial semantics: negative length
    ArgumentError, closed-stream IOError before the len==0 shortcut, len==0
    returns the cleared buffer, EOF (file or closed-write pipe) raises EOFError,
    and the output buffer is filled and returned with its encoding preserved. The
    pipe read view now folds in new bytes incrementally so #ungetc/#ungetbyte
    survive a refresh. (io_getpartial, rb_io_readpartial; readpartial_spec.rb)
  • IO.copy_stream — transfers through real IO (File/pipe/StringIO), path /
    #to_path, or duck-typed #readpartial/#read(len,buf) sources, to IO / path /
    #to_path / #write destinations; a File destination is flushed to disk
    immediately; a src_offset preads a descriptor-backed source without moving its
    position (rejected for a StringIO); a zero copy_length is a no-op.
    (copy_stream; copy_stream_spec.rb + fixtures/classes.rb)

CI gates

  • go build ./... and GOWORK=off go test ./... pass (pre-existing 127.0.0.1:1
    network tests excepted).
  • 100% statement coverage on POSIX for every function introduced or reworked
    (whitebox io_wave22_test.go, asserted byte-for-byte vs MRI 4.0.5;
    rbgo-specific fallbacks — StringIO write not transcoded, synchronous-pipe
    would-block — are noted as such).
  • gofmt -l internal/vm/ empty; wasip1 build passes; scoped -race on the
    changed code is clean.

Remaining core/io failures are outside this file set: subprocess IO.popen
duplex streams, IO::Buffer mmap (map/open_fixture), set_encoding(nil,nil)
default-snapshot semantics (would require reworking the lazy encoding model),
Tempfile-backed copy_stream, and missing Errno::ESPIPE / IO::WaitReadable
constants.

🤖 Generated with Claude Code

tannevaled and others added 6 commits September 9, 2026 15:56
…n (core/io +20)

IO.pipe (spawn.go): apply the read end's external/internal encoding from the
call's encoding arguments (Encoding, name, "ext:int", BOM|, or #to_str), else
the Encoding defaults captured at creation; the write end carries none; the
receiver class is honoured so a subclass's .pipe yields subclass instances.
Ref: ruby/spec core/io/pipe_spec.rb; MRI io.c rb_io_s_pipe / pipe_pair_close.

IO#write (io.go defIOWrite): an all-empty write returns 0 before any
closed/writable check (io.c io_write returns before rb_io_check_writable when
the string is empty), so write("") on a read-only or closed stream no longer
raises; a non-empty write still does. When the stream has an explicit non-BINARY
external encoding, each written String is transcoded from its own encoding to it
(rb_str_encode / io.c do_writeconv + make_writeconv), returning the transcoded
byte count and raising Encoding::UndefinedConversionError /
InvalidByteSequenceError on unconvertible bytes; StringIO and BINARY/unset
streams write bytes unchanged, and the argument String is never mutated.
Ref: ruby/spec core/io/write_spec.rb + shared/write.rb (:io_write_transcode).

Encoding options (io_class.go): a leading "BOM|" marker on an :encoding /
:external_encoding / :internal_encoding option value is stripped before the
encoding is resolved (rb_io_extract_encoding_option records the BOM as a flag),
so File.open(..., encoding: "bom|utf-8") no longer raises.
Ref: ruby/spec core/io/write_spec.rb "ignores the 'bom|' prefix".

core/io ruby/spec: pass 1149 -> 1169 (fail 161 -> 142, error 119 -> 118).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…umerator (core/io +11)

gets/readline/readlines/each_line limit (io.go resolveGetsArgs) and IO#sysseek
offset (io_descriptors.go) now coerce through ioOfftArg, so a Bignum too large
for the C off_t raises RangeError instead of TypeError (io.c rb_io_getline_1 /
sysseek NUM2OFFT). IO#sysseek also raises IOError on a closed stream before
coercing the offset.
Ref: ruby/spec core/io/{gets,sysseek}_spec.rb, shared/pos.rb, shared/readlines.rb
("does not accept Integers that don't fit in a C off_t").

IO.foreach (io_class.go): each yield updates $. to the current line number and
$_ is cleared to nil once the last line is read (io.c rb_io_s_foreach's
gets-based loop); with no block it returns an Enumerator whose #size is nil
(the line count is unknown without reading) rather than an eagerly built array.
Ref: ruby/spec core/io/foreach_spec.rb ("updates $. with each yield",
"sets $_ to nil", "returned Enumerator size should return nil").

TestIOForeachReadlines updated to the MRI-verified Enumerator behaviour
(to_a.length 3, size nil, class Enumerator).

core/io ruby/spec: pass 1169 -> 1180 (fail 142 -> 131).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…artial 4->14)

readpartial (spawn.go) is no longer aliased to read_nonblock. It now follows
io.c io_getpartial / rb_io_readpartial: a negative maxlen raises ArgumentError;
a closed or read-shut stream raises IOError before the maxlen==0 shortcut;
maxlen==0 returns the (cleared) output buffer immediately without blocking; a
normal call returns at most maxlen buffered bytes and, at EOF (write end closed,
no bytes), clears the output buffer then raises EOFError. The optional output
buffer receives the bytes and is returned by identity with its encoding
preserved (ioReadResult).
Ref: ruby/spec core/io/readpartial_spec.rb.

pipeRefresh (io.go) now folds only the newly written pipe bytes into the
reader's buf (tracked by pipeSynced) instead of re-snapshotting the whole shared
buffer, so bytes pushed back with #ungetc/#ungetbyte — spliced into buf ahead of
the read cursor — survive a refresh and are read back before fresh pipe data.
This fixes readpartial-after-ungetc on a pipe.

core/io ruby/spec: pass 1180 -> 1191 (fail 131 -> 123, error 118 -> 115).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… copy_stream 29->53)

Rewrote IO.copy_stream (io_class.go copyStreamRead/copyStreamWrite) to io.c
copy_stream semantics:
  - a source or destination that is a real IO (File/pipe/StringIO) transfers
    through it: the source is read from its current position (advancing it), and
    the destination's buffer is flushed so a File reaches disk immediately (as
    copy_stream's descriptor-level write does) — fixing the empty File.read and
    ENOENT results where writes previously stayed buffered;
  - a src_offset preads the source at that offset WITHOUT moving its position,
    and is accepted for a descriptor-backed File but still rejected for a
    StringIO ("cannot specify src_offset for non-IO");
  - a String / #to_path argument names a file to read (with length/offset) or a
    file to truncate-write;
  - a bare object is drained through #readpartial (preferred, until EOFError) or
    #read(len, buf), and written through #write;
  - a zero copy_length transfers nothing and calls neither #read nor #write.
Ref: ruby/spec core/io/copy_stream_spec.rb + fixtures/classes.rb.

IO#readpartial (spawn.go): at end-of-data a regular stream (File/StringIO) now
raises EOFError like a closed-write pipe, instead of the pipe-only EAGAIN — so
readpartial-draining a File to EOF terminates correctly.

Remaining copy_stream failures are Tempfile-backed (stdlib) and a missing
Errno::ESPIPE constant, both outside this file set.

core/io ruby/spec: pass 1191 -> 1215 (fail 123 -> 108, error 115 -> 106).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ng divergence

Correctness fix: a "BOM|" marker is accepted only on the :encoding option (and
mode strings), NOT on :external_encoding / :internal_encoding — MRI resolves the
latter as plain encoding names and raises "unknown encoding name" for
external_encoding: "bom|utf-8". The earlier encNameFromOpt wrongly stripped BOM|
there; reverted those two option paths to encodingArg and removed the helper.
Verified File.open(external_encoding: "bom|utf-8") now raises like MRI while
File.open(encoding: "bom|utf-8") still works.

Tests (io_wave22_test.go, whitebox, package vm) bring every function introduced
or reworked this wave to 100% statement coverage, each asserted byte-for-byte
against MRI Ruby 4.0.5 (rbgo-specific behaviour — StringIO write not transcoded,
readpartial/write_nonblock would-block on the synchronous pipe model — is noted
as such): IO.pipe encoding resolution (encPairFromArgs/stripBOMPrefix), write
transcoding + empty-write + non-String coercion (ioWriteAll/ioWriteEncode/
asWriteString), IO.copy_stream over IO/path/#to_path/duck-typed #read/
#readpartial sources and IO/path/#to_path/#write destinations with offset and
zero-length guards (copyStreamRead/copyStreamReadObject/copyStreamWrite), the
:encoding/:external_encoding BOM handling (extractEncodingOption), write_nonblock,
readpartial guards, and the gets/sysseek off_t RangeError.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@tannevaled
tannevaled merged commit 09e1ab1 into main Sep 9, 2026
7 checks passed
@tannevaled
tannevaled deleted the wave22/io branch September 9, 2026 20:46
tannevaled added a commit that referenced this pull request Sep 10, 2026
…606)

First run past 20,000 passing ruby/spec examples.

Wave-22 merged three bibliography-first conformance PRs:
- #605 core/io: IO.pipe encodings, IO#write/write_nonblock transcoding, bom|
  handling, off_t RangeError, IO.foreach $./$_, IO#readpartial, IO.copy_stream
  with duck-typed sources and destinations — +77.
- #604 core/kernel: Kernel#putc (new), #p/#print/#puts forwarding through
  $stdout so a mocked stream observes them, #p rendering per MRI rb_inspect — +15.
- #603 VM: OpOpenSingletonClass now resets defaultVis, so a private in one
  'class << self' body no longer leaks into a later reopened one — +4.

Measured total on merged main: 20093 passing. Local measure at TIMEOUT=25;
CI runs TIMEOUT=60 so its total is >= this floor.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant