fix: repair mojibake corruption; restore RUF001/002/003 tripwire - #8
Merged
Conversation
The prior CI cleanup wrongly ignored RUF001/002/003 as "intentional typography." They were correctly flagging mojibake — UTF-8 text decoded as cp1252 and saved back — including two user-facing strings in cli.py that crash `memorywire --help` on native Windows consoles (the emitted byte can't be decoded, the capture thread dies). This is live in v0.5.0 on PyPI. - Repair mojibake across all .py (src, tests, ui, scripts, examples) with ftfy.fix_encoding — restores the intended em dashes, ellipses, curly quotes, and section signs. Char-level only; no behaviour change. - Remove the RUF001/002/003 ignore so the rules resume catching corruption. - extract_abstract.py legitimately emits em/en dashes (LaTeX dash conversion) — annotate with a scoped `# noqa: RUF001`, the correct surgical exception rather than a blanket ignore. - Delete the untracked scratch scripts/inspect_newamp.py. Verified against the CI dependency set (optional extras absent): ruff check + ruff format --check + `mypy src/` all clean; tests/unit/test_smoke.py::test_cli_help_exits_zero now passes; full unit suite 396 green. Docs (*.md, ~800 sequences) still carry mojibake — not ruff-checked and non-shipping — tracked as a follow-up. v0.5.0 on PyPI still ships the corrupted cli.py; a patch release is needed to fix it for installed users.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug I introduced, and the real bug it hid
The earlier CI-green PR ignored
RUF001/002/003as "intentional typography." That was wrong — those rules were correctly flagging mojibake (UTF-8 decoded as cp1252 and saved back). Two of the corrupted spots are user-facing incli.py:src/memorywire/cli.py:55— the--helpdescription (— memorywire CLI), printed on everymemorywire --help.src/memorywire/cli.py:262— the recall-output truncation marker (…).On a native Windows console the corrupted byte can't be encoded, the capture thread dies, and
memorywire --helpreturnsNonestdout — the failingtests/unit/test_smoke.py::test_cli_help_exits_zero. This is live in v0.5.0 on PyPI. (CI's Windows runners use UTF-8, so CI didn't catch it — real users do.)Fix
.py(src,tests,ui,scripts,examples) withftfy.fix_encoding— restores the intended em dashes / ellipses / curly quotes / section signs. Char-level only; verified no behaviour change.RUF001/002/003ignore so the rules resume catching corruption.scripts/extract_abstract.pylegitimately emits em/en dashes (LaTeX dash conversion) — annotated with a scoped# noqa: RUF001, the correct surgical exception.scripts/inspect_newamp.py.Verification (CI dependency set — optional extras absent)
ruff check .— All checks passedruff format --check .— cleanmypy src/— Success (also withlangchain/mcpuninstalled)tests/unit/test_smoke.py::test_cli_help_exits_zero— now passesFollow-ups (not in this PR)
docs/*.mdstill carry ~800 mojibake sequences (worst:memorywire-paper.md,adapters.md). Not ruff-checked, non-shipping — safe to fix next. The arXiv.tex, README, and CITATION.cff are already clean.cli.py— a patch release (v0.5.1) is needed to fix it for installed users.