Repository navigation
io: read-side transcoding, File.open Hash options, IO#set_encoding_by_bom, BOM reads (core/io +62, core/file +8) - #596
Merged
Merged
Conversation
… set_encoding_by_bom (core/io +57)
Honour the external/internal encoding on the read side, and treat a trailing
Hash to File.open/File.new/Kernel#open as options (io.c rb_scan_args "12:")
rather than a mode String:
- File.open/new/Kernel#open resolve mode + :encoding/:external_encoding/
:internal_encoding through ioResolveModeEnc and record the encoding on the
stream (previously a Hash mode raised "no implicit conversion of {..} into
String" and a "r:ext:int" string set no encoding at all).
- IO#read with no length transcodes external→internal and tags the result
(io.c io_enc_str); read(size) stays BINARY; a passed buffer is retagged only
on a full read. A pure-ASCII line is retagged without a converter (MRI's
identical-bytes fast path), so e.g. IBM866→UTF-8 of ASCII does not raise.
- IO#readchar decodes one character of the external encoding (multi-byte
EUC-JP/UTF-8) via decodeCharFrom, then transcodes to the internal encoding.
- IO#gets family tags/transcodes each line the same way (rb_io_getline_1).
- New IO#set_encoding_by_bom (rb_io_set_encoding_by_bom + io_strip_bom): strip
a UTF-8/16/32 BOM and set the external encoding, with the binmode /
encoding-already-set / conversion-set ArgumentErrors and the not-readable nil.
References read: ruby v3_4_0 io.c (rb_io_extract_modeenc, io_enc_str,
rb_io_getline_1, rb_io_set_encoding_by_bom, io_strip_bom) and ruby/spec
core/io/{read,readchar,gets,set_encoding_by_bom}_spec.rb.
core/io: 1028 -> 1085 passing (+57).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…paths - IO.read/File.read with a "rb:BOM|enc" mode strip a leading UTF-8/16/32 BOM from the file start (io.c), sharing detectBOM with set_encoding_by_bom. - Add io_wave20_test.go covering every new branch to 100%: File.open Hash options + mode coercion (vmodeString), read transcoding and buffer retagging, readchar external-encoding decoding (incl. invalid/truncated lead bytes), the gets encoding family, set_encoding_by_bom (all BOMs, nil and error paths), and the IO.read BOM-mode stripping. Asserted against MRI Ruby 4.0.5. - Drop an unreachable readchar length clamp (decodeCharFrom never returns more bytes than remain) so the branch set stays fully covered. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
File.open/File.new/Kernel#open no longer call fileMode (they resolve the mode through openFileArgs/vmodeString), leaving IO.sysopen its only caller — and sysopen always passes a mode, so fileMode's default branch became dead code. Inline the resolution into sysopen via vmodeString (which also coerces an Integer flag set or a #to_str mode object, unlike the old strArg) and delete fileMode. Also drop File.open's redundant zero-arg guard: openFileArgs already rejects a missing path with the same ArgumentError. Keeps the touched files at 100% statement coverage. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
tannevaled
added a commit
that referenced
this pull request
Sep 8, 2026
…598) Wave-20 merged three bibliography-first conformance PRs: - #596 core/io: File.open/new trailing-Hash options, IO#read/readchar/gets external->internal transcoding, IO#set_encoding_by_bom, BOM stripping — +62 (and core/file +8 incidental). - #597 core/enumerator: Enumerator::Product class, Chain/ArithmeticSequence constructors, Enumerator#take_while break-early (clearing the produce_spec hang) — +49. - #595 core/module: extend_object, const_source_location, const_set const_added + redefinition warning, class_variable ancestry search, Module.constants singleton, remove/undef_method hooks — +19. Measured total on merged main: 19915 passing (2172 files load, +1 that was a hang). Local measure at TIMEOUT=25; CI runs TIMEOUT=60 so its total is >= this floor. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Wave-20 IO conformance, focused on the read side of encoding and on File.open's
argument handling. All against MRI Ruby 4.0.5, byte-for-byte.
rb_scan_args "12:"+rb_io_extract_modeenc). A Hash mode argument used toraise
no implicit conversion of {..} into String, and a"r:ext:int"modestring set no encoding at all; now
mode:/:encoding/:external_encoding/
:internal_encodingare resolved through the existingioResolveModeEncandrecorded on the stream. Integer and
#to_int/#to_strmode objects arecoerced (
vmodeString).(io.c
io_enc_str);read(size)stays BINARY; a passed buffer is retaggedonly on a full read, and left untouched on a sized read. Pure-ASCII content is
retagged without a converter (MRI's identical-bytes fast path), so e.g.
IBM866→UTF-8 of an ASCII line does not raise.
EUC-JP/UTF-8 via
decodeCharFrom) and transcodes to the internal encoding —previously it always assumed UTF-8 and returned U+FFFD for legacy bytes.
rb_io_getline_1), including the BINARY-external suppression of the internalencoding.
rb_io_set_encoding_by_bom+io_strip_bom): strip a UTF-8/16/32 BOM and set the external encoding, withthe binmode / encoding-already-set / conversion-set
ArgumentErrors, theclosed
IOError, and the not-readablenil."rb:BOM|enc"mode strips a leading BOM from the filestart (shared
detectBOM).IO.sysopen's mode now also goes throughvmodeString; the now-unusedfileModehelper is deleted.References read (per method)
ruby v3_4_0
io.c:rb_io_extract_modeenc,rb_scan_args "12:",io_enc_str,rb_io_getline_1,io_getc,rb_io_set_encoding_by_bom,io_strip_bom;ruby/spec
core/io/{read,readchar,gets,set_encoding_by_bom}_spec.rbandcore/filefor the shared File.open paths.Measured conformance (ruby/spec, MRI-verified)
Per file: read_spec 72→105, readchar_spec 6→14, gets_spec 32→36,
set_encoding_by_bom_spec 0→16.
Gates
go build ./...andGOWORK=off go test ./...— green (full vm suite passes).internal/vm/io.go,io_class.go,io_descriptors.go): all 89 functions at 100.0%; new whiteboxio_wave20_test.gocovers every added branch, asserted against MRI 4.0.5.gofmt -l internal/vm/empty;GOOS=wasip1 GOARCH=wasm go build ./...green.🤖 Generated with Claude Code