Fix inverted null-count check in inclusive metrics NotStartsWith - #3891
Open
dylanpulver wants to merge 1 commit into
Open
Fix inverted null-count check in inclusive metrics NotStartsWith#3891dylanpulver wants to merge 1 commit into
dylanpulver wants to merge 1 commit into
Conversation
`_InclusiveMetricsEvaluationVisitor._may_contain_null` inverted the condition from the Java implementation. It reported "may contain null" when a null count was present (including a count of 0) and "cannot contain null" when the count was absent. Both directions are wrong. A file with a proven null count of 0 was never pruned by `NotStartsWith`, and a file whose null count is unknown could be pruned even though its nulls satisfy the predicate, silently dropping matching rows. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
dylanpulver
force-pushed
the
fix-inclusive-metrics-may-contain-null
branch
from
September 1, 2026 15:25
3dce8e9 to
5c9e0af
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
_InclusiveMetricsEvaluationVisitor._may_contain_nullis inverted relative to Java.visitors.py:1203returnsself.null_counts is None or (field_id in self.null_counts and self.null_counts.get(field_id) is not None).null_countsis set toEMPTY_DICTin_MetricsEvaluationVisitor.__init__, so the first disjunct is dead and the rest means "may contain null iff a count is present" — backwards on both branches.Java,
api/src/main/java/org/apache/iceberg/expressions/InclusiveMetricsEvaluator.java:100-102:0The only caller is
visit_not_starts_with(visitors.py:1451), which uses it to guard the bounds-pruning path. The first row makes that pruning dead code. The second row prunes a file whose null count is unknown, which drops rows that do match — the row-level evaluator returnsTrueforNotStartsWithon a NULL (visitors.py:523-524).I have not traced whether PyIceberg's own writer emits bounds without null counts, so treat the missed-pruning half as always reachable and the row-dropping half as reachable for any spec-legal manifest but not demonstrated end to end here.
Are these changes tested?
tests/expressions/test_evaluator.py::test_string_not_starts_withis a port of Java'stestStringNotStartsWithminusFILE_5, the one fixture that reaches the pruning branch. All 12 of its assertions areassert should_read; there are none of the other polarity, so a function that always answers "may contain null" cannot fail it.Added
data_file_5(mirrors Java'sFILE_5:null_value_counts={3: 0}, boundsabc..abcdefghi) anddata_file_6(same,null_value_counts=None), plus three assertions.Full suite is
4001 passed, 3 skippedon both58749a3and this branch — the new assertions live inside an existing test, so the count does not move and the mutants are the evidence. Reverting the source with the tests kept fails ondata_file_5(assert not True). The naive fixself.null_counts.get(field_id, 0) != 0gets the zero-count case right and still fails ondata_file_6, which is why that second fixture is there.make lintpasses all 12 hooks. Integration tests were not run.Are there any user-facing changes?
A
NotStartsWithscan now skips data files that have a zero null count and bounds entirely inside the prefix, and stops skipping files whose null count is unknown. Fewer files read in the first case, more in the second.Written with AI assistance (Claude Opus 4.8). The Java references above were read from
apache/icebergmain, and the measurements were run against this branch.