Skip to content

fix(pipeline): read a BOM-marked requirements.txt as pip does (#2941) - #2996

Open
b84330808 wants to merge 1 commit into
DeusData:mainfrom
b84330808:fix-dep-manifest-bom-utf16
Open

b84330808 wants to merge 1 commit into
DeusData:mainfrom
b84330808:fix-dep-manifest-bom-utf16

Conversation

@b84330808

Copy link
Copy Markdown

What does this PR do?

Fixes #2941.

pip reads requirements.txt by its BOM, and Windows tools write BOMs. Windows PowerShell 5.1 saves pip freeze > requirements.txt as UTF-16 LE with a BOM and CRLF, and many editors save UTF-8 with a BOM. The dependency-manifest reader in pass_k8s.c took the raw bytes:

  • UTF-16: the line parser stops at the NUL after the first letter, so the file gives one Package named \xff\xfe + r (the @bytes:fffe72 in the issue) and none of the real dependencies.
  • UTF-8 with a BOM: the BOM sticks to the first package name. Filippo-Venturini/ctxvault, the repo named in the issue, has this form on its default branch today (EF BB BF before annotated-doc==0.0.4).

The fix decodes the manifest once, right after k8s_read_file in k8s_prep_file. It drops a UTF-8 BOM, and it transcodes UTF-16 LE/BE with a BOM to UTF-8 (surrogate pairs combined, an unpaired surrogate becomes U+FFFD). go.mod goes through the same reader, so it gets the same handling. A file without a BOM is read exactly as before.

Tests, in tests/test_grammar_probe_f.c next to the go.mod group. Each indexes a project with only requirements.txt (requests==2.32.3, uvicorn==0.30.0, CRLF) and checks the sorted Package names:

  • probe_requirements_utf16le_bom_issue2941: UTF-16 LE with a BOM, the PowerShell 5.1 case.
  • probe_requirements_utf16be_bom_issue2941: UTF-16 BE with a BOM.
  • probe_requirements_utf8_bom_issue2941: UTF-8 with a BOM.
  • probe_requirements_utf8_plain_issue2941: plain UTF-8, unchanged behaviour.

How I ran them: on ubuntu-24.04 (GitHub Actions on my fork), make -f Makefile.cbm test-focused TEST_SUITES=grammar_probe_f gives 188 passed, and scripts/test.sh ends with All tests passed. With pass_k8s.c reverted to main and the new tests kept, the suite gives 185 passed, 3 failed: the three BOM tests (e.g. "uvicorn,requests" != "requests,uvicorn" for the UTF-8 BOM case); the plain UTF-8 test passes either way.

Locally (Windows, no make or zlib) I compiled pass_k8s.c and the test file with zig cc 0.16 (clang 20, -target x86_64-linux-gnu, the Makefile's -Wall -Wextra -Werror set): both clean. I also ran the decoder and parse_requirements_deps, copied out of pass_k8s.c into a small harness built with UBSan:

input main this branch
UTF-16 LE + BOM \xff\xfe r requests,uvicorn
UTF-16 BE + BOM \xfe\xff requests,uvicorn
UTF-8 + BOM \xef\xbb\xbf requests,uvicorn requests,uvicorn
UTF-8 requests,uvicorn requests,uvicorn

pass_k8s.c has no clang-format 20.1.8 drift. I haven't run cppcheck or clang-tidy.

I used Claude Code to write this change.

Checklist

  • Every commit is signed off (git commit -s) — required, CI rejects
    unsigned commits (DCO, see CONTRIBUTING.md)
  • Tests pass locally (make -f Makefile.cbm test)
  • Lint passes (make -f Makefile.cbm lint-ci)
  • New behavior is covered by a test (reproduce-first for bug fixes)

…ta#2941)

Windows PowerShell 5.1 saves `pip freeze > requirements.txt` as UTF-16 LE
with a BOM, and many Windows editors save UTF-8 with a BOM. pip decodes
the file by its BOM, but the dependency-manifest reader in pass_k8s took
the raw bytes. UTF-16 ended at the NUL after the first letter, so the file
gave one bogus Package ("\xff\xfe" + "r") and none of its dependencies,
and a UTF-8 BOM stuck to the first package name.

Decode the manifest once after reading it: drop a UTF-8 BOM, and transcode
UTF-16 LE/BE with a BOM to UTF-8. go.mod goes through the same reader.
Files without a BOM are read exactly as before.

Signed-off-by: Wei-Lun Li <b84330808@gmail.com>
@b84330808
b84330808 requested a review from DeusData as a code owner November 14, 2026 18:31
@github-actions

Copy link
Copy Markdown

Thanks for opening this — it has been seen, and it is queued.

This note is automated, but it is not a brush-off: it exists so you know where your PR stands instead of having to guess from silence.

Current review status: working through a backlog. 0.9.1-rc.1 is out, so the release freeze that held reviews is over — but it left a large queue of open pull requests behind it, and we are reading through them oldest-first. The background is in discussion #1144.

What that means for this PR, concretely:

  • It will not be closed for inactivity. No stale bot touches pull requests here.
  • It may still sit a while before a human reads it. That is on us, not on you.
  • Older PRs are read first, so a recent one is not being skipped — it is behind a queue.

Things that will genuinely speed it up whenever review does happen:

  • Keep it rebased on main — the tree is moving quickly right now, and a conflicting branch cannot be reviewed as the diff you intended.
  • Get CI green, or say which failures you believe are pre-existing.
  • Keep the change to one claim. Bundled features and refactors get split before they get merged, which costs you a round trip.
  • Every commit needs a sign-off (git commit -s) — CI enforces DCO.

If this fixes a bug, a reproduction we can run is worth more than a description of the symptom.

Thanks for contributing, and sorry in advance for the wait.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

requirements.txt saved as UTF-16 (or with a BOM) indexes as one bogus Package and none of its dependencies

1 participant