compare: keep column names with '..' as flat keys for dictdiffer#51
Open
HrachShah wants to merge 2 commits into
Open
compare: keep column names with '..' as flat keys for dictdiffer#51HrachShah wants to merge 2 commits into
HrachShah wants to merge 2 commits into
Conversation
added 2 commits
July 4, 2026 03:41
The csv module's underlying C parser caps each field at 131072 bytes by default. load_csv had no protection against that, so a row with a long string (e.g. a nucleotide sequence, a large JSON blob, an embedded log line) raised an uncaught _csv.Error instead of being returned to the caller. Set csv.field_size_limit(sys.maxsize) at the top of load_csv so the parser is allowed to handle the longest field the caller can possibly give it. sys.maxsize is the platform's PY_SSIZE_T_MAX and matches the recommendation in the Python csv docs for "set the limit as high as possible" without overflowing the parser. Closes simonw#41.
dictdiffer's default dot_notation=True interprets '..' in a key as a path with a parent step, so a column named 'name..date_range' gets parsed as if it were nested under a 'name' key. When a row sits in both previous and current but the column has been renamed from 'name.date_range' to 'name..date_range', the resulting diff contains 'add' and 'remove' 2-tuples for the column swap instead of 'change' 3-tuples, and the existing per-row unpack in compare() raises 'ValueError: not enough values to unpack (expected 2, got 1)'. compare() now passes dot_notation=False so dictdiffer treats every top-level column name as a flat string. The 'ignore_columns' lookup is also affected - with dot_notation=True a column named '..foo' is never matched by the ignore list because dictdiffer sees it as a path, not a literal key. The per-row unpack is unchanged; with dot_notation=False the path is always a list, so the existing 'field[0] if isinstance(field, list) else field' flattening still turns it back into the right column name. Closes simonw#40.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #40
dictdiffer's default
dot_notation=Trueinterprets..in a key as a path with a parent step. When a column is namedname..date_range, dictdiffer parses it as a nested path rather than a literal key, and emitsadd/remove2-tuples for column renames instead ofchange3-tuples. The existing per-row unpack incompare()then raisesValueError: not enough values to unpack (expected 2, got 1).This PR passes
dot_notation=Falseto thediff()call so every top-level column name is treated as a flat string. Theignore_columnslookup is also affected: withdot_notation=Truea column named..foois never matched by the ignore list because dictdiffer sees it as a path, not a literal key. The per-row unpack incompare()is unchanged; withdot_notation=Falsethe path is always a list, so the existingfield[0] if isinstance(field, list) else fieldflattening still turns it back into the right column name.Reproducer (from the issue):
Pre-fix:
ValueError: not enough values to unpack (expected 2, got 1).Post-fix: returns the expected diff with
name..date_rangelisted undercolumns_addedandname.date_rangeundercolumns_removed.Includes two new tests in
tests/test_csv_diff.py:test_compare_handles_double_dot_in_column_name— the column-rename casetest_compare_handles_double_dot_in_column_value_change— value change in a..column, which already worked (since the result is still achange3-tuple) but was unwrapped with a fragile field-string code path; now unwrapped via the list path the new option forces.All 28 tests pass.