Repository navigation
Encoding subsystem: dynamic find names, macCyrillic/UTF8-MAC codecs, Converter output-buffer carryover (MRI 4.0.6) - #570
Merged
Merged
Conversation
…back errinfo
Advance MRI 4.0.6 conformance in the encoding subsystem:
- Encoding.find("external"/"filesystem"/"locale") now resolves to the current
Encoding.default_external, and "internal" to Encoding.default_internal (nil
when unset), tracking the setters instead of the pinned UTF-8 aliases.
- Encoding.default_external= rejects nil with ArgumentError ("default external
can not be nil") rather than the generic coercion TypeError, and stores the
default on the VM so find() reflects it.
- Add the macCyrillic (Macintosh Cyrillic) codec via golang.org/x/text so
String#encode and Encoding::Converter can transcode UTF-8<->macCyrillic.
- Encoding::Converter#putback no longer overwrites the read-again bytes that
#primitive_errinfo reports, matching MRI where putback and primitive_errinfo
are independent.
Specs: default_external 5->8, find 11->12 (error cleared), converter/convert
5->7 (2 errors cleared), converter/putback 6->7.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…estination - #primitive_convert now holds converted output that does not fit the destination (MRI's internal output-buffer carryover) in converterObj.pendingOut and flushes it on the next call. stepLoop returns the overflow bytes and, when a character does not fit the byte cap, writes the prefix that fits, holds the remainder and consumes the character — so a full destination returns :destination_buffer_full with the source consumed, and successive capped calls accumulate correctly (the "aabbb" offset behaviour). - Add UTF8-MAC (HFS+ NFD) as a Converter destination: characters are encoded in canonical decomposition via golang.org/x/text/unicode/norm, so UTF-8 -> UTF8-MAC transcoding (and its invalid-byte handling) works. Specs: converter/primitive_convert 27p/3f/2e -> 32p/0f/0e (offset carryover, two destination_buffer_full source-clearing cases, two UTF8-MAC cases). Encoding cluster now 596 pass / 3 fail / 0 error. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add primitive_convert carryover cases where the held output exceeds a later call's byte cap (exercising the room clamp and the produced>size hold), and a Shift_JIS undefined-conversion case exercising encodeCharTo's encoder-error path. All asserted against MRI Ruby 4.0.6. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Advance MRI 4.0.6 conformance in the encoding subsystem (
core/encoding/*andcore/encoding/converter/*). The encoding spec cluster goes from 584 pass / 15 fail+error to 596 pass / 3 fail / 0 error — 12 examples fixed, no regressions.Fixes
Encoding class
Encoding.find("external"/"filesystem"/"locale")now resolves to the currentEncoding.default_external, and"internal"toEncoding.default_internal(nilwhen unset), tracking the setters instead of the pinned UTF-8 aliases.Encoding.default_external=rejectsnilwithArgumentError("default external can not be nil") instead of the generic coercionTypeError, and stores the default on the VM sofind()reflects it.Codecs
golang.org/x/text/encoding/charmap, soString#encodeandEncoding::Convertercan transcode UTF-8 ↔ macCyrillic.Converterdestination: characters are encoded in canonical decomposition viagolang.org/x/text/unicode/norm, so UTF-8 → UTF8-MAC transcoding and its invalid-byte handling work.Encoding::Converter engine
#primitive_convertnow models MRI's internal output-buffer carryover: converted output that does not fit the destination is held in the converter and flushed on the next call. A character that does not fit the byte cap has its fitting prefix written, the remainder held, and the character consumed — so a full destination returns:destination_buffer_fullwith the source consumed, and successive capped calls accumulate correctly (the"aabbb"offset behaviour, and the香→ISO-2022-JPdestination_buffer_fullsource-clearing cases).#putbackno longer overwrites the read-again bytes that#primitive_errinforeports, matching MRI where the two are independent.Spec deltas
encoding/default_externalencoding/findconverter/convertconverter/putbackconverter/primitive_convertOut of scope (documented residuals)
encoding/compatible(1):Symbol#to_symfrom a non-ASCII string does not track its source encoding — needs per-symbol encoding storage in the object model.converter/finish(1): ISO-2022-JP shift-state (\e(Breset) deferral requires stateful multi-hop encoding.converter/last_error(1):éISO-8859-1→Big5 —golang.org/x/text's Big5 table maps a character MRI leaves undefined.Verification
go test ./...green;go test ./internal/vm/green;GOOS=js GOARCH=wasmbuild OK;gofmt -lempty.internal/vm/encoding_residuals_wave13_test.go(asserted against MRI Ruby 4.0.6).🤖 Generated with Claude Code