Repository navigation
Conversation
|
|
… sets using a sorted value set Since apacheGH-46777 is_in predicates with more than 50 values are not simplified against row group statistics at all, because filtering the value set and rebinding the call for every guarantee was prohibitively expensive. Large value sets are now simplified with a binary search instead: - The is_in kernel init keeps a copy of the value set sorted in the order of the comparison kernels, with nulls removed. An already sorted value set is reused as is after a single O(N) check; an unsorted one is sorted lazily and thread-safely on the first simplification, so eager kernel calls never pay for sorting. Value sets containing NaN are not searchable, since statistics cannot bound NaN. - SimplifyWithGuarantee applies all inequality guarantees on a field to a large is_in call at once, narrowing an index range into the sorted value set with lower/upper bound searches. If the range is empty the call simplifies to false (subject to the null matching behavior), otherwise it is left unchanged. This is exact for any number of values, costs O(log N) per guarantee and never rebinds. Applying both bounds together is what allows pruning a row group inside the hull of the value set but holding none of its values, which no per-bound check can do. Value sets up to 50 values keep using the existing filter-and-rebind path.
…d of via sort_indices + take Sorting a copy of the values with std::sort is about 14x faster than the generic sort_indices + take path for a shuffled set of 10M int64 values (0.19s vs 2.6s). Nulls are dropped while copying and a set containing NaN is reported as not searchable before sorting, since NaN would break the strict weak ordering std::sort requires. Non-primitive types (binary, decimal) keep using the generic path.
…in is_in pruning Extract nullable min/max bounds from Parquet statistics for large value sets. Respect GetNullMatchingBehavior() in both simplification paths and when rebinding narrowed sets. Add expression and Parquet regressions for nullable ranges and conflicting deprecated skip_nulls options.
…and prune nullable ranges in every simplification path Inequality guarantees are now restated through the order preserving casts wrapping a filter's argument before they are applied: strict bounds are weakened to inclusive ones and the bound is put through the same casts, since e.g. f64 = 51.00000001 satisfies f64 > 51 but casts to 51f. This applies to plain comparisons and small is_in value sets as well as the large value set path, which previously was the only one handling it. The nullable range guarantee produced by Parquet statistics for row groups with nulls, (x >= min and x <= max) or is_null(x), is now extracted for all simplifications rather than only for large is_in value sets, and an is_valid(x) member of the guarantee rules out nulls for the inequalities on x.
rok
force-pushed
the
GH-55292-is-in-row-group-pruning
branch
from
November 3, 2026 17:12
a3f317c to
79750a7
Compare
…L and INCONCLUSIVE calls - Defer concatenating, checking and sorting the value set to the first GetSortedValueSet() call (std::call_once), so plain is_in kernel calls pay nothing extra at init. - Simplify an is_in call without possible matches to its exact result for every null matching behavior: invert(true_unless_null(arg)) for EMIT_NULL and INCONCLUSIVE, or a null literal for INCONCLUSIVE with a null in the value set, so these prune row groups too. - Factor out Comparison::IsOrderPreservingCast, derive IsSearchableValueSetType from the accessor trait, and drop unreachable NaN, type and comparison checks. - Test chunked value sets and the new null matching results.
…pruning and condense tests - Sort the value set with sort_indices + take and search it with search_sorted instead of hand-written visitors. Decimal and fixed size binary value sets are no longer searchable since search_sorted does not support them. - Make the large is_in simplification tests table-driven, merge the null matching tests, and share a helper which checks that simplified filters evaluate like the originals. - Merge the two Parquet row group pruning tests into one file whose row groups contain nulls.
…and max instead of sorting them Sorting a value set costs 1.5-3x as much as building the is_in lookup table, so for 10M shuffled ids the one-time sort outweighed the row groups it pruned and scans got slower than without pruning. Value sets longer than kIsInMaxSortedValueSet (2^17) are now only bounded by their smallest and largest value (min_max), which is cheap at any size and prunes as well when the values are clustered.
…ith a cast MSVC flags constructing a std::vector<int32_t> from int64_t iterators as a narrowing conversion (C4244), which fails the Windows build.
…ered alike and keep NaN in small is_in value sets A guarantee bound of another type was brought into the field's type whenever a safe cast round tripped. Strings and numbers round trip but are ordered differently, so the statistics of a string column read as int64 pruned row groups holding matches of a large is_in. Bounds are now only converted between types which are ordered alike. Value sets of up to 50 values containing NaN are no longer filtered against inequality guarantees: statistics cannot bound NaN, which is_in matches.
…ge and simplify against per-field ranges Expression simplification no longer reaches into the is_in kernel state. A kernel state may implement the new internal RangeEvaluator interface to tell what its call evaluates to when the argument lies in a range; is_in implements it with its lazily sorted value set and its null matching rules, and scalar_set_lookup_internal.h is removed. SimplifyWithGuarantee folds the guarantee once into the inequalities and nullability of each field and simplifies every call in a single pass. Guarantee bounds are converted by one rule (Inequality::Restate) for comparisons, small is_in value sets and range evaluation alike: a bound of another type must be ordered like the field's type and convert to it exactly.
…st the ones which matter Removing each piece of the simplification in turn showed that the NaN check on guarantee bounds in expression.cc duplicates the kernel's, and that decimal bounds never converted to an integer or floating point field's type; both are removed. Guards which no test covered now have one: NaN in a sorted value set and in a value set bounded by its smallest and largest value, a strict bound next to an inclusive one at the same value, date bounds on a timestamp field, and index_in never being simplified.
… instead of ignoring them A bound of another type than the field's (the statistics of a float64 column read as float32, say) was ignored unless it converted exactly, so comparisons against such statistics were no longer simplified. The bound is now cast safely and, unless it round trips, strict comparisons are weakened to inclusive ones: 0.2 rounds up to 0.2f, which a stored 0.2 is read as. Also apply clang-format.
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The simplification compares uncast double entries against a rounded bound, potentially reducing a matching filter to false and incorrectly pruning rows.
1 open issue
What changed in this PR
Restores Parquet row-group pruning for large is_in filters without repeatedly filtering and rebinding value sets, while extending simplification to nullable statistics.
Changes:
- Adds kernel range evaluation with lazy sorted-set lookup and min/max bounds for very large sets.
- Combines per-field bounds and handles nullable guarantees, cast rounding, and null-matching options.
- Adds expression-equivalence and Parquet row-group pruning tests.
| File | Description |
|---|---|
cpp/src/arrow/dataset/file_parquet_test.cc |
Tests large-set and nullable row-group pruning. |
cpp/src/arrow/compute/range_evaluation_internal.h |
Defines the internal kernel range-evaluation interface. |
cpp/src/arrow/compute/kernels/scalar_set_lookup.cc |
Adds cached value-set range evaluation for is_in. |
cpp/src/arrow/compute/expression.cc |
Integrates per-field ranges and cast-aware simplification. |
cpp/src/arrow/compute/expression_test.cc |
Covers ranges, coercion, null behavior, and execution equivalence. |
cpp/src/arrow/compute/expression_internal.h |
Refines order-preserving cast detection to exclude wrapping casts. |
Review effort: Balanced
| std::string func_name = Comparison::GetName(cmp); | ||
| DCHECK_NE(func_name, "na"); | ||
| std::vector<Datum> args{options->value_set, guarantee.bound}; | ||
| std::vector<Datum> args{options->value_set, bound}; |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
Since #46859,
is_inpredicates with more than 50 values are not simplified against Parquet row group statistics at all, so such filters read every row group. In addition,SimplifyWithGuaranteedoes not recognize the statistics guarantee of a row group that may contain nulls,(x >= min and x <= max) or is_null(x), so no filter prunes those row groups. See #55292 for the background. Theis_inproblem was reported and fixed in Polars in pola-rs/polars#29923.What changes are included in this PR?
RangeEvaluator(range_evaluation_internal.h), lets a kernel state tell what its call evaluates to when the argument is known to lie in a range: unknown, false, null, or false unless the argument is null.SimplifyWithGuaranteediscovers it withdynamic_caston the kernel state and replaces the call when the result is known. The expression code does not know the semantics of the function or the layout of its state.is_inimplements it with a sorted value set. For value sets of an integer, floating point, temporal, duration, string or binary type, theis_inkernel state keeps a reference to the value set and sorts it (sort_indices+take, nulls dropped) the first time it is evaluated over a range; plain kernel calls pay nothing extra. The range is looked up withsearch_sorted, O(log N) per guarantee and with no rebinding. If no value can match, the result follows the null matching behavior:false,invert(true_unless_null(x))forEMIT_NULLandINCONCLUSIVEon nullable data, or a null literal forINCONCLUSIVEwith a null in the value set. Sorting costs 1.5 to 3 times as much as building the kernel's lookup table, so value sets longer than 2^17 are only bounded by their smallest and largest value (min_max), which is cheap at any size and prunes as well when the values are clustered. Value sets containing NaN are not searched, since statistics cannot bound NaN.SimplifyWithGuaranteefolds the guarantee once into the inequalities and nullability of each field and simplifies every call against the range of its field in a single pass, instead of one pass per inequality. Using both row group bounds together is what lets us skip row groups that lie inside the hull of the value set but hold none of its values. Value sets of up to 50 values keep using the existing filter-and-rebind path; longer ones are evaluated by the kernel.(cmp(x, a) and cmp(x, b)) or is_null(x)now yield both inequalities, and anis_valid(x)guarantee rules out nulls. Comparisons and smallis_insets now prune row groups with nulls too.Inequality::Restate), used by comparisons, smallis_insets and range evaluation alike:is_insimplification: givenf64 > 51,cast(f64, float32) == 51was simplified tofalsealthoughf64 = 51.00000001casts to51.Comparison::StripOrderPreservingCastsno longer strips same-width unsigned to signed casts that may wrap (allow_int_overflow).is_inpath now honors the deprecatedskip_nullsoption (GetNullMatchingBehavior()) and leaves value sets containing NaN alone, like the large one.Are these changes tested?
Yes. New expression tests:
SimplifyIsInLargeValueSet,SimplifyIsInLargeValueSetNullMatching(which also checks that simplified filters evaluate like the originals on rows with nulls),SimplifyWithNullableRangeGuarantee,SimplifyWithBoundOfAnotherType,SimplifyComparisonThroughNonInjectiveCast,SimplifyIsInDeprecatedSkipNullsandSimplifyLargeIsInThenExecute. A new Parquet dataset test checks which row groups are read for largeis_infilters under every null matching behavior, and for comparisons on row groups with nulls.Scan timings on a 49.3M row file with 47 row groups of 2^20 rows and a single sorted int64
idcolumn (release build on macOS arm64, file in page cache, median of 5Scanner::ToTablecalls after a warm-up, each binding a fresh filter). The value sets are shuffled distinct ids drawn from the given row groups:is_invalue setThe 1M ids in two distant row groups show the limit of bounding value sets longer than 2^17 by their smallest and largest value: the 19 row groups in between are read too.
Are there any user-facing changes?
No API changes.
is_infilters with more than 50 values prune Parquet row groups again (except for decimal, fixed size binary and view value sets, and dictionary columns). Range filters and smallis_infilters now also prune row groups that contain nulls.Fixes to existing simplification:
falseand skip matching rows; they no longer are.is_invalue sets containing NaN could be simplified tofalsefor a row group holding NaN; they are now left unchanged.NotImplementedduring row group pruning; its statistics are now ignored.Was AI used for this PR?
PR code and description written by:
Reviewed before submission by:
The code was drafted with an AI assistant; I reviewed, tested and benchmarked it and iterated on the design.