Repository navigation
fix(corekit): decode as UTF-8 when a StringField's charset rejects 'replace' (#1706) - #1728
Merged
Merged
Conversation
…eplace' (#1706) - StringField.post_process retried a failed decode with 'replace' outside any try, so a codec that raises under 'replace' too let UnicodeError escape unpack: idna (strict only) on every Python, and punycode on non-ASCII octets before 3.13. - The retry now falls back once more, to UTF-8 with 'replace', which cannot raise; the existing lossless check returns a DecodedString, so the field packs back the captured octets. - The errors docstring no longer says such a codec still raises. Tests: TestStringFieldLossless.test_a_codec_that_rejects_replace_does_not_raise (4 subtests fail on origin/main on 3.14, 7 on 3.8); the lossless module passes on 3.8, 3.11, 3.14 and 3.15. Closes #1706
Owner
Author
|
Verdict: GOOD TO GO at
|
Contributor
|
Coverage: 89.35% (unit tier, Python 3.15,
Per-file detail: the |
This was referenced Oct 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Please follow the guide below
You will be asked some questions, please read them carefully and answer honestly
Put an
xinto all the boxes [ ] relevant to your pull request (like that [x])Use Preview tab to see how your pull request will actually look like
Searched for similar pull requests
Followed the coding style (
make pylint,make mypy,make isort)make testpasses, and a test case covers the change. Narrow runs only, listed below.Added a changelog entry under
docs/source/changelog/and regeneratedCHANGELOG.md, if the change is user-visible: N/A — centralised in docs(changelog): shared 1.5.0 changelog — long-lived, merges last (#610, #616, #617, #618, #620) #657What is the purpose of your pull request?
Tick the commit type your subject line carries.
fix— corrects a defectfeat— adds a featureperf— changes performance, not behaviourrefactor— changes neither behaviour nor performancetest— tests onlydocs— documentation onlyci— workflows or build toolingrelease— bumps the version or rolls up a distributionchore— anything elseDescription of your pull request and other information
Closes #1706.
errors='replace'outside anytry. A codec that also raises under'replace'therefore let aUnicodeErrorescapeunpack. That isidnaon every Python,undefined, andpunycodeon non-ASCII octets before 3.13.'replace', which cannot raise. UTF-8 is the fallback encoding the docstring already names, and the one theunquotebranch already uses. The existing lossless check returns aDecodedString, so packing gives back the captured octets byte for byte. Theerrorsdocstring no longer says such a codec still raises.errors='ignore',idnaonb'xn--caf-dma'now reads as'xn--caf-dma', while'strict'still gives'café'.test_a_codec_that_rejects_replace_does_not_raisefails onmain(4 subtests on 3.14, 7 errors on 3.8) and passes on 3.8, 3.11, 3.14 and 3.15.encodings(111 or 112) under five error schemes, over 165 byte samples, on 3.8, 3.11 and 3.14. That found 0 exceptions and 0 round-trip mismatches.