Skip to content

fix(csv-parse): preserve data when trimming single-byte encodings - #518

Open
PDGGK wants to merge 1 commit into
adaltas:masterfrom
PDGGK:fix/csv-trim-single-byte-encoding
Open

PDGGK wants to merge 1 commit into
adaltas:masterfrom
PDGGK:fix/csv-trim-single-byte-encoding

Conversation

@PDGGK

@PDGGK PDGGK commented Oct 2, 2026

Copy link
Copy Markdown

Problem

The whitespace table added in #483 can strip ordinary field bytes with ASCII and Latin-1 encodings. Node truncates Unicode code points when encoding these characters: U+205F becomes _, U+202F becomes /, and other whitespace characters collide with ordinary data.

For example, parsing _alice,/home,(draft) with ASCII encoding and trimming changes the field values to alice,home,draft).

Change

Build on #483's whitespace-table design by retaining a whitespace sequence only when it round-trips through the configured encoding. This preserves representable whitespace, encoding aliases, and the existing UTF-8/UTF-16 whitespace table without changing parser state handling.

This does not claim byte-for-byte parity with String.trim() for arbitrary raw high-bit ASCII bytes such as 0xA0. Existing buffer-output mode with trim: true remains outside this fix.

The patch contains source and regression tests only, following #483. Generated distributions were rebuilt and checked; they are not included as release artifacts.

Validation

  • All 27 added native tests pass; reverting only production code gives 24 failures and 3 passes with the same tests.
  • Full workspace tests pass on Node 20.20.2 and 24.19.0: all 14 test projects, 1,104 passing tests and 6 pending. Node 22 was not separately tested.
  • Full workspace build passes; lint has no errors and 6 existing pending-test warnings.
  • Exact-output checks pass for source ESM, rebuilt CJS/browser ESM, encoding aliases, and stream splits: 13,469 assertions on each tested Node version.

@PDGGK
PDGGK marked this pull request as ready for review October 3, 2026 06:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant