Skip to content

Encoding compatibility: Converter.new not-found, concat encoding negotiation, multibyte codepoints, ascii_only? - #530

Merged
tannevaled merged 4 commits into
mainfrom
feat/agent-encsys
Aug 31, 2026
Merged

tannevaled merged 4 commits into
mainfrom
feat/agent-encsys

Conversation

@tannevaled

Copy link
Copy Markdown
Contributor

Advances MRI 4.0.6 conformance of the string-encoding compatibility subsystem.

Changes

  • Encoding::Converter.new now matches MRI's rb_econv_open: an unknown
    encoding name, an identical src/dst, or a known-but-undeliverable pair (any
    pair through the dummy UTF-7) raises Encoding::ConverterNotFoundError
    ("code converter not found (SRC to DST)", unresolvable names kept verbatim,
    resolvable ones canonicalised) instead of ArgumentError or silently
    constructing. A non-String, non-Encoding argument still raises TypeError.
    This unblocks encode residuals that expect the correct error class.

  • String#concat / #<< with an aliased receiver (a.concat(a, a)) now
    contributes a's INITIAL value at each position ("hellohellohello"), by
    freezing aliased arguments before the append loop mutates the receiver.

  • Integer codepoint against a multibyte encoding (String#concat/#<< and
    Integer#chr): the value is interpreted as a big-endian byte sequence that
    must form exactly one valid character (MRI rb_enc_mbcput + precise-mbclen).
    A valid multibyte codepoint (e.g. 0xa4a2 in EUC-JP) is now accepted where
    rbgo previously raised "out of char range"; a lone lead byte (0x81 in
    EUC-JP/Shift_JIS) now raises "invalid codepoint 0x81 in EUC-JP" instead of
    silently appending the byte. Single-byte encodings keep the 0..255 rule.

  • String#ascii_only? now reports MRI's ENC_CODERANGE_7BIT: content in a
    non-ASCII-compatible encoding (UTF-16/32, …) is never 7-bit, even empty or
    all-ASCII in bytes.

Spec deltas (rubyspec)

  • core/string/concat_spec: 27→29 (both failures fixed)
  • core/string/ascii_only_spec: 11→13 (both failures fixed)
  • core/encoding/converter/new_spec: 19/19
  • core/string/{plus,append,multiply}_spec, core/encoding/compatible_spec,
    force_encoding_spec, b_spec: no regressions

Out of scope

  • %c / sprintf encoding negotiation lives in the external
    github.com/go-ruby-format/format package (and format_bind.go), not editable here.
  • The compatible_spec Symbol case needs per-Symbol encoding tracking
    (object.Symbol is a plain string); valid_encoding? residuals need codecs
    for Big5/CP949/EUC-KR/… — both outside this cluster.

Verification

  • go test ./internal/vm/ and go test ./... green (redis-dependent bindings
    log connection noise but pass).
  • wasm build (GOOS=js GOARCH=wasm) OK; gofmt -l clean on changed files.
  • New internal/vm/encoding_compat_test.go (package vm_test) keeps every
    changed function at 100% coverage.

🤖 Generated with Claude Code

tannevaled and others added 4 commits August 31, 2026 15:38
…oncat

Encoding::Converter.new now matches MRI's rb_econv_open: an unknown encoding
name, or a known but undeliverable pair (e.g. any pair involving the dummy
UTF-7), raises Encoding::ConverterNotFoundError with a "code converter not
found (SRC to DST)" message (unresolvable names kept verbatim, resolvable ones
canonicalised) instead of ArgumentError or silently constructing. A
non-String, non-Encoding argument still raises TypeError.

String#concat/<< with an argument that aliases the receiver (a.concat(a, a))
now contributes a's INITIAL value at every position ("hellohellohello"), by
freezing aliased arguments before the append loop mutates the receiver.

Two converter_test cases that asserted the old (non-MRI) behaviour are
corrected to expect ConverterNotFoundError at construction.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…/chr

String#concat/<< and Integer#chr with an Integer argument now interpret the
value as a big-endian byte sequence that must form exactly one valid character
in a legacy multibyte encoding (EUC-JP, Shift_JIS, …), matching MRI's
rb_enc_mbcput + precise-mbclen check:
  - a valid multibyte codepoint (e.g. 0xa4a2 in EUC-JP) is now accepted and
    unpacked to its bytes, where before rbgo wrongly raised "out of char range";
  - a lone lead byte such as 0x81 in EUC-JP/Shift_JIS is in byte range but is
    not a character, so it now raises "invalid codepoint 0x81 in EUC-JP" instead
    of silently appending the byte.
Single-byte encodings (ISO-8859-1, BINARY, …) keep the plain 0..255 rule.

New encoding_compat_test.go (package vm_test) covers Converter.new not-found /
type-error / valid-pair paths, the aliased-receiver concat snapshot, and the
multibyte codepoint validation.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds encoding_compat_test cases so every branch of the new converterEncName
(#to_str resolving to a known and to an unknown name) and bigEndianBytes
(codepoint 0) is exercised, keeping the changed functions at 100%.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
#ascii_only? reports MRI's ENC_CODERANGE_7BIT, not a bare byte scan: a string
tagged with a non-ASCII-compatible encoding (UTF-16/32, …) is never 7-bit even
when empty or all-ASCII in bytes, so it now returns false there instead of
scanning only the bytes. ASCII-compatible encodings keep the byte-scan result.

Fixes the two ascii_only_spec examples for empty and non-empty strings in a
non-ASCII-compatible encoding.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@tannevaled
tannevaled merged commit fa966b4 into main Aug 31, 2026
7 checks passed
@tannevaled
tannevaled deleted the feat/agent-encsys branch August 31, 2026 14:55
tannevaled added a commit that referenced this pull request Aug 31, 2026
Locks in the gains from #530 (Encoding compatibility: Converter.new,
concat negotiation, multibyte codepoints, ascii_only?), #531 (language
def/method/super/keyword-argument), #532 (Enumerable residuals) and #533
(Array/Hash value equality dispatching #==/#eql?/#hash). Measured 17654
passing on the pinned corpus, up from 17470.

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant