Skip to content

io: read-side transcoding, File.open Hash options, IO#set_encoding_by_bom, BOM reads (core/io +62, core/file +8) - #596

Merged
tannevaled merged 3 commits into
mainfrom
wave20/io
Sep 8, 2026
Merged

tannevaled merged 3 commits into
mainfrom
wave20/io

Conversation

@tannevaled

Copy link
Copy Markdown
Contributor

Summary

Wave-20 IO conformance, focused on the read side of encoding and on File.open's
argument handling. All against MRI Ruby 4.0.5, byte-for-byte.

  • File.open / File.new / Kernel#open honour a trailing Hash as options (io.c
    rb_scan_args "12:" + rb_io_extract_modeenc). A Hash mode argument used to
    raise no implicit conversion of {..} into String, and a "r:ext:int" mode
    string set no encoding at all; now mode: / :encoding / :external_encoding
    / :internal_encoding are resolved through the existing ioResolveModeEnc and
    recorded on the stream. Integer and #to_int/#to_str mode objects are
    coerced (vmodeString).
  • IO#read transcodes external→internal on a full read and tags the result
    (io.c io_enc_str); read(size) stays BINARY; a passed buffer is retagged
    only on a full read, and left untouched on a sized read. Pure-ASCII content is
    retagged without a converter (MRI's identical-bytes fast path), so e.g.
    IBM866→UTF-8 of an ASCII line does not raise.
  • IO#readchar decodes one character of the external encoding (multi-byte
    EUC-JP/UTF-8 via decodeCharFrom) and transcodes to the internal encoding —
    previously it always assumed UTF-8 and returned U+FFFD for legacy bytes.
  • IO#gets family tags/transcodes each line the same way (io.c
    rb_io_getline_1), including the BINARY-external suppression of the internal
    encoding.
  • New IO#set_encoding_by_bom (io.c rb_io_set_encoding_by_bom +
    io_strip_bom): strip a UTF-8/16/32 BOM and set the external encoding, with
    the binmode / encoding-already-set / conversion-set ArgumentErrors, the
    closed IOError, and the not-readable nil.
  • IO.read / File.read "rb:BOM|enc" mode strips a leading BOM from the file
    start (shared detectBOM).

IO.sysopen's mode now also goes through vmodeString; the now-unused
fileMode helper is deleted.

References read (per method)

ruby v3_4_0 io.c: rb_io_extract_modeenc, rb_scan_args "12:", io_enc_str,
rb_io_getline_1, io_getc, rb_io_set_encoding_by_bom, io_strip_bom;
ruby/spec core/io/{read,readchar,gets,set_encoding_by_bom}_spec.rb and
core/file for the shared File.open paths.

Measured conformance (ruby/spec, MRI-verified)

suite before after Δ
core/io 1028 1090 +62
core/file 601 609 +8 (File.open encoding, incidental)
library/stringio 582 582 0 (no regression)

Per file: read_spec 72→105, readchar_spec 6→14, gets_spec 32→36,
set_encoding_by_bom_spec 0→16.

Gates

  • go build ./... and GOWORK=off go test ./... — green (full vm suite passes).
  • 100% statement coverage on the touched files (internal/vm/io.go,
    io_class.go, io_descriptors.go): all 89 functions at 100.0%; new whitebox
    io_wave20_test.go covers every added branch, asserted against MRI 4.0.5.
  • gofmt -l internal/vm/ empty; GOOS=wasip1 GOARCH=wasm go build ./... green.

🤖 Generated with Claude Code

tannevaled and others added 3 commits September 8, 2026 12:13
… set_encoding_by_bom (core/io +57)

Honour the external/internal encoding on the read side, and treat a trailing
Hash to File.open/File.new/Kernel#open as options (io.c rb_scan_args "12:")
rather than a mode String:

- File.open/new/Kernel#open resolve mode + :encoding/:external_encoding/
  :internal_encoding through ioResolveModeEnc and record the encoding on the
  stream (previously a Hash mode raised "no implicit conversion of {..} into
  String" and a "r:ext:int" string set no encoding at all).
- IO#read with no length transcodes external→internal and tags the result
  (io.c io_enc_str); read(size) stays BINARY; a passed buffer is retagged only
  on a full read. A pure-ASCII line is retagged without a converter (MRI's
  identical-bytes fast path), so e.g. IBM866→UTF-8 of ASCII does not raise.
- IO#readchar decodes one character of the external encoding (multi-byte
  EUC-JP/UTF-8) via decodeCharFrom, then transcodes to the internal encoding.
- IO#gets family tags/transcodes each line the same way (rb_io_getline_1).
- New IO#set_encoding_by_bom (rb_io_set_encoding_by_bom + io_strip_bom): strip
  a UTF-8/16/32 BOM and set the external encoding, with the binmode /
  encoding-already-set / conversion-set ArgumentErrors and the not-readable nil.

References read: ruby v3_4_0 io.c (rb_io_extract_modeenc, io_enc_str,
rb_io_getline_1, rb_io_set_encoding_by_bom, io_strip_bom) and ruby/spec
core/io/{read,readchar,gets,set_encoding_by_bom}_spec.rb.

core/io: 1028 -> 1085 passing (+57).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…paths

- IO.read/File.read with a "rb:BOM|enc" mode strip a leading UTF-8/16/32 BOM
  from the file start (io.c), sharing detectBOM with set_encoding_by_bom.
- Add io_wave20_test.go covering every new branch to 100%: File.open Hash
  options + mode coercion (vmodeString), read transcoding and buffer retagging,
  readchar external-encoding decoding (incl. invalid/truncated lead bytes),
  the gets encoding family, set_encoding_by_bom (all BOMs, nil and error
  paths), and the IO.read BOM-mode stripping. Asserted against MRI Ruby 4.0.5.
- Drop an unreachable readchar length clamp (decodeCharFrom never returns more
  bytes than remain) so the branch set stays fully covered.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
File.open/File.new/Kernel#open no longer call fileMode (they resolve the mode
through openFileArgs/vmodeString), leaving IO.sysopen its only caller — and
sysopen always passes a mode, so fileMode's default branch became dead code.
Inline the resolution into sysopen via vmodeString (which also coerces an
Integer flag set or a #to_str mode object, unlike the old strArg) and delete
fileMode. Also drop File.open's redundant zero-arg guard: openFileArgs already
rejects a missing path with the same ArgumentError. Keeps the touched files at
100% statement coverage.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@tannevaled
tannevaled merged commit d0911f9 into main Sep 8, 2026
7 checks passed
@tannevaled
tannevaled deleted the wave20/io branch September 8, 2026 11:58
tannevaled added a commit that referenced this pull request Sep 8, 2026
…598)

Wave-20 merged three bibliography-first conformance PRs:
- #596 core/io: File.open/new trailing-Hash options, IO#read/readchar/gets
  external->internal transcoding, IO#set_encoding_by_bom, BOM stripping — +62
  (and core/file +8 incidental).
- #597 core/enumerator: Enumerator::Product class, Chain/ArithmeticSequence
  constructors, Enumerator#take_while break-early (clearing the produce_spec
  hang) — +49.
- #595 core/module: extend_object, const_source_location, const_set const_added
  + redefinition warning, class_variable ancestry search, Module.constants
  singleton, remove/undef_method hooks — +19.

Measured total on merged main: 19915 passing (2172 files load, +1 that was a
hang). Local measure at TIMEOUT=25; CI runs TIMEOUT=60 so its total is >= this floor.

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant