Repository navigation
Encoding compatibility: Converter.new not-found, concat encoding negotiation, multibyte codepoints, ascii_only? - #530
Merged
Merged
Conversation
…oncat
Encoding::Converter.new now matches MRI's rb_econv_open: an unknown encoding
name, or a known but undeliverable pair (e.g. any pair involving the dummy
UTF-7), raises Encoding::ConverterNotFoundError with a "code converter not
found (SRC to DST)" message (unresolvable names kept verbatim, resolvable ones
canonicalised) instead of ArgumentError or silently constructing. A
non-String, non-Encoding argument still raises TypeError.
String#concat/<< with an argument that aliases the receiver (a.concat(a, a))
now contributes a's INITIAL value at every position ("hellohellohello"), by
freezing aliased arguments before the append loop mutates the receiver.
Two converter_test cases that asserted the old (non-MRI) behaviour are
corrected to expect ConverterNotFoundError at construction.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…/chr
String#concat/<< and Integer#chr with an Integer argument now interpret the
value as a big-endian byte sequence that must form exactly one valid character
in a legacy multibyte encoding (EUC-JP, Shift_JIS, …), matching MRI's
rb_enc_mbcput + precise-mbclen check:
- a valid multibyte codepoint (e.g. 0xa4a2 in EUC-JP) is now accepted and
unpacked to its bytes, where before rbgo wrongly raised "out of char range";
- a lone lead byte such as 0x81 in EUC-JP/Shift_JIS is in byte range but is
not a character, so it now raises "invalid codepoint 0x81 in EUC-JP" instead
of silently appending the byte.
Single-byte encodings (ISO-8859-1, BINARY, …) keep the plain 0..255 rule.
New encoding_compat_test.go (package vm_test) covers Converter.new not-found /
type-error / valid-pair paths, the aliased-receiver concat snapshot, and the
multibyte codepoint validation.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds encoding_compat_test cases so every branch of the new converterEncName (#to_str resolving to a known and to an unknown name) and bigEndianBytes (codepoint 0) is exercised, keeping the changed functions at 100%. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
#ascii_only? reports MRI's ENC_CODERANGE_7BIT, not a bare byte scan: a string tagged with a non-ASCII-compatible encoding (UTF-16/32, …) is never 7-bit even when empty or all-ASCII in bytes, so it now returns false there instead of scanning only the bytes. ASCII-compatible encodings keep the byte-scan result. Fixes the two ascii_only_spec examples for empty and non-empty strings in a non-ASCII-compatible encoding. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
tannevaled
added a commit
that referenced
this pull request
Aug 31, 2026
Locks in the gains from #530 (Encoding compatibility: Converter.new, concat negotiation, multibyte codepoints, ascii_only?), #531 (language def/method/super/keyword-argument), #532 (Enumerable residuals) and #533 (Array/Hash value equality dispatching #==/#eql?/#hash). Measured 17654 passing on the pinned corpus, up from 17470. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Advances MRI 4.0.6 conformance of the string-encoding compatibility subsystem.
Changes
Encoding::Converter.newnow matches MRI'srb_econv_open: an unknownencoding name, an identical src/dst, or a known-but-undeliverable pair (any
pair through the dummy UTF-7) raises
Encoding::ConverterNotFoundError("code converter not found (SRC to DST)", unresolvable names kept verbatim,
resolvable ones canonicalised) instead of
ArgumentErroror silentlyconstructing. A non-String, non-Encoding argument still raises
TypeError.This unblocks encode residuals that expect the correct error class.
String#concat/#<<with an aliased receiver (a.concat(a, a)) nowcontributes
a's INITIAL value at each position ("hellohellohello"), byfreezing aliased arguments before the append loop mutates the receiver.
Integer codepoint against a multibyte encoding (
String#concat/#<<andInteger#chr): the value is interpreted as a big-endian byte sequence thatmust form exactly one valid character (MRI
rb_enc_mbcput+ precise-mbclen).A valid multibyte codepoint (e.g.
0xa4a2in EUC-JP) is now accepted whererbgo previously raised "out of char range"; a lone lead byte (
0x81inEUC-JP/Shift_JIS) now raises "invalid codepoint 0x81 in EUC-JP" instead of
silently appending the byte. Single-byte encodings keep the 0..255 rule.
String#ascii_only?now reports MRI'sENC_CODERANGE_7BIT: content in anon-ASCII-compatible encoding (UTF-16/32, …) is never 7-bit, even empty or
all-ASCII in bytes.
Spec deltas (rubyspec)
core/string/concat_spec: 27→29 (both failures fixed)core/string/ascii_only_spec: 11→13 (both failures fixed)core/encoding/converter/new_spec: 19/19core/string/{plus,append,multiply}_spec,core/encoding/compatible_spec,force_encoding_spec,b_spec: no regressionsOut of scope
%c/ sprintf encoding negotiation lives in the externalgithub.com/go-ruby-format/formatpackage (andformat_bind.go), not editable here.compatible_specSymbol case needs per-Symbol encoding tracking(
object.Symbolis a plain string);valid_encoding?residuals need codecsfor Big5/CP949/EUC-KR/… — both outside this cluster.
Verification
go test ./internal/vm/andgo test ./...green (redis-dependent bindingslog connection noise but pass).
GOOS=js GOARCH=wasm) OK;gofmt -lclean on changed files.internal/vm/encoding_compat_test.go(packagevm_test) keeps everychanged function at 100% coverage.
🤖 Generated with Claude Code