[Radim Hopp (rhopp)]

LWLP-4918: Preserve non-UTF-8 Python sources during version rewrites

Summary

lightwell-source-to-sdist no longer fails when a package’s default version-metadata scan encounters a Python source file that is not valid UTF-8. PEP 263 sources, including pylint’s Latin-1 test fixture, can now be scanned and rewritten without corrupting their original non-UTF-8 bytes.

Problem

The default version rewrite previously read every .py file with strict UTF-8 decoding:

path.read_text(encoding="utf-8")

That caused deterministic failures for packages that intentionally ship non-UTF-8 Python fixtures, such as:

tests/functional/i/implicit/implicit_str_concat_latin1.py

The failing task reported:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 105: invalid continuation byte

This blocked all affected Python cumulative builds at source-to-sdist.

Root cause

_plan_default_rewrites() scans every .py file under the source tree and previously decoded each one as strict UTF-8. Python source files are not required to be UTF-8; PEP 263 permits other source encodings.

Fix

• Default Python-source scans now read and write with errors="surrogateescape".
• Universal-newline handling is retained, so CR-only and CRLF Python sources are still normalized exactly as before.
• The planner now carries the Python-source policy with each planned rewrite, so a symlinked .py file whose resolved target has a different suffix uses the same read and write policy.
• Authoritative metadata files (pyproject.toml, setup.cfg, PKG-INFO, METADATA) continue to require strict UTF-8.
• Explicit rewrite rules continue to require strict UTF-8 and still fail closed.

Testing

Added regression coverage for:

• A non-UTF-8 Python source containing __version__
• A symlinked .py file whose resolved target has a different suffix
• CR-only and CRLF Python sources
• Explicit rewrite rules targeting non-UTF-8 Python files
• A non-matching non-UTF-8 Python file remaining byte-identical

Validated with:

pytest --cov --cov-report=term-missing
ruff check .
ruff format --check .
mypy src
uv build

Results:

• 289 tests passed
• Coverage: 93.28%
• Ruff lint/format passed
• Mypy passed
• Package build passed

Issue

LWLP-4918