Skip to content

Encoding subsystem: dynamic find names, macCyrillic/UTF8-MAC codecs, Converter output-buffer carryover (MRI 4.0.6) - #570

Merged
tannevaled merged 3 commits into
mainfrom
feat/agent-encoding
Sep 6, 2026
Merged

tannevaled merged 3 commits into
mainfrom
feat/agent-encoding

Conversation

@tannevaled

Copy link
Copy Markdown
Contributor

Advance MRI 4.0.6 conformance in the encoding subsystem (core/encoding/* and core/encoding/converter/*). The encoding spec cluster goes from 584 pass / 15 fail+error to 596 pass / 3 fail / 0 error — 12 examples fixed, no regressions.

Fixes

Encoding class

  • Encoding.find("external"/"filesystem"/"locale") now resolves to the current Encoding.default_external, and "internal" to Encoding.default_internal (nil when unset), tracking the setters instead of the pinned UTF-8 aliases.
  • Encoding.default_external= rejects nil with ArgumentError ("default external can not be nil") instead of the generic coercion TypeError, and stores the default on the VM so find() reflects it.

Codecs

  • Add the macCyrillic (Macintosh Cyrillic) codec via golang.org/x/text/encoding/charmap, so String#encode and Encoding::Converter can transcode UTF-8 ↔ macCyrillic.
  • Add UTF8-MAC (HFS+ NFD) as a Converter destination: characters are encoded in canonical decomposition via golang.org/x/text/unicode/norm, so UTF-8 → UTF8-MAC transcoding and its invalid-byte handling work.

Encoding::Converter engine

  • #primitive_convert now models MRI's internal output-buffer carryover: converted output that does not fit the destination is held in the converter and flushed on the next call. A character that does not fit the byte cap has its fitting prefix written, the remainder held, and the character consumed — so a full destination returns :destination_buffer_full with the source consumed, and successive capped calls accumulate correctly (the "aabbb" offset behaviour, and the 香→ISO-2022-JP destination_buffer_full source-clearing cases).
  • #putback no longer overwrites the read-again bytes that #primitive_errinfo reports, matching MRI where the two are independent.

Spec deltas

spec before after
encoding/default_external 5p/3f 8p/0f
encoding/find 11p/1e 12p/0e
converter/convert 5p/2e 7p/0e
converter/putback 6p/1f 7p/0f
converter/primitive_convert 27p/3f/2e 32p/0f/0e

Out of scope (documented residuals)

  • encoding/compatible (1): Symbol#to_sym from a non-ASCII string does not track its source encoding — needs per-symbol encoding storage in the object model.
  • converter/finish (1): ISO-2022-JP shift-state (\e(B reset) deferral requires stateful multi-hop encoding.
  • converter/last_error (1): é ISO-8859-1→Big5 — golang.org/x/text's Big5 table maps a character MRI leaves undefined.

Verification

  • go test ./... green; go test ./internal/vm/ green; GOOS=js GOARCH=wasm build OK; gofmt -l empty.
  • 100% merged coverage on the changed functions; every new branch/helper exercised by internal/vm/encoding_residuals_wave13_test.go (asserted against MRI Ruby 4.0.6).

🤖 Generated with Claude Code

tannevaled and others added 3 commits September 6, 2026 10:15
…back errinfo

Advance MRI 4.0.6 conformance in the encoding subsystem:

- Encoding.find("external"/"filesystem"/"locale") now resolves to the current
  Encoding.default_external, and "internal" to Encoding.default_internal (nil
  when unset), tracking the setters instead of the pinned UTF-8 aliases.
- Encoding.default_external= rejects nil with ArgumentError ("default external
  can not be nil") rather than the generic coercion TypeError, and stores the
  default on the VM so find() reflects it.
- Add the macCyrillic (Macintosh Cyrillic) codec via golang.org/x/text so
  String#encode and Encoding::Converter can transcode UTF-8<->macCyrillic.
- Encoding::Converter#putback no longer overwrites the read-again bytes that
  #primitive_errinfo reports, matching MRI where putback and primitive_errinfo
  are independent.

Specs: default_external 5->8, find 11->12 (error cleared), converter/convert
5->7 (2 errors cleared), converter/putback 6->7.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…estination

- #primitive_convert now holds converted output that does not fit the
  destination (MRI's internal output-buffer carryover) in converterObj.pendingOut
  and flushes it on the next call. stepLoop returns the overflow bytes and, when a
  character does not fit the byte cap, writes the prefix that fits, holds the
  remainder and consumes the character — so a full destination returns
  :destination_buffer_full with the source consumed, and successive capped calls
  accumulate correctly (the "aabbb" offset behaviour).
- Add UTF8-MAC (HFS+ NFD) as a Converter destination: characters are encoded in
  canonical decomposition via golang.org/x/text/unicode/norm, so UTF-8 -> UTF8-MAC
  transcoding (and its invalid-byte handling) works.

Specs: converter/primitive_convert 27p/3f/2e -> 32p/0f/0e (offset carryover, two
destination_buffer_full source-clearing cases, two UTF8-MAC cases). Encoding
cluster now 596 pass / 3 fail / 0 error.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add primitive_convert carryover cases where the held output exceeds a later
call's byte cap (exercising the room clamp and the produced>size hold), and a
Shift_JIS undefined-conversion case exercising encodeCharTo's encoder-error
path. All asserted against MRI Ruby 4.0.6.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@tannevaled
tannevaled merged commit 26924d9 into main Sep 6, 2026
7 checks passed
@tannevaled
tannevaled deleted the feat/agent-encoding branch September 6, 2026 09:28
tannevaled added a commit that referenced this pull request Sep 6, 2026
Locks in #569 (Hash), #570 (Encoding subsystem), #571 (Range) and #572 (the
!= opcode dispatching a user-defined #!=). Measured 18790 passing on the pinned
corpus, up from 18767.

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant