[shrivasshankar]

Which issue does this PR close?

• Closes Estimate Decimal filter selectivity #25613.

Rationale for this change

Decimal columns were excluded from interval analysis, so any filter touching
one fell back to the default 20% selectivity. For the Q6-style predicate in the
issue, that meant estimating ~1.2M rows when only ~114K match.

What changes are included in this PR?

• FilterExec statistics use a new check_statistics_support, which also
  accepts Decimal32/64/128/256.
• next_value / prev_value handle decimals, so strict comparisons get tight
  bounds (x < 24.00 becomes x <= 23.99). At the limit of a decimal's
  precision the bound becomes unbounded instead of overflowing.
• get_extreme_value! supports Decimal32/64, which previously hit
  unreachable!().
• CastExpr now propagates constraints back through lossless decimal widening
  casts. Without this, a literal with a different scale than the column (e.g.
  a >= 0.055 on Decimal(15, 2), coerced to Decimal128(30, 15)) left the
  column range untouched and the estimate at 100%.
• Decimal arithmetic (+ - * / %) is not analyzed: it changes the precision
  and scale of its operands, so those predicates keep the default selectivity.

check_support and is_datatype_supported are unchanged, so symmetric hash
join pruning behaves as before. Decimal support there could be a follow-up.

What is the testing strategy for this PR?

• Unit tests for decimal next/prev values across all four widths, including
  precision boundaries like 9.99 / -9.99 for Decimal(3, 2), plus strict
  comparison bounds and Decimal32/64 overflow handling.
• test_filter_statistics_decimal_expr and
  test_filter_statistics_decimal_mixed_scale_expr check row estimates and
  tightened min/max for same-scale and mixed-scale predicates. The mixed-scale
  test mirrors the plan the SQL planner produces.
• test_filter_statistics_decimal_arithmetic_expr checks that decimal
  multiplication and subtraction predicates use the default selectivity and
  keep the column's own type.
• filter_statistics_decimal.slt runs Q6-style ranges with fractional
  literals on a parquet table with min/max stats, checking EXPLAIN estimates
  and query results, including a mixed-scale literal and decimal arithmetic.
• New cases in test_cast_constraint_propagation, plus boundary checks that
  propagating through a decimal widening cast never excludes a valid input,
  covering cross-width casts, a negative scale and the precision limits.
• test_custom_filter_selectivity relied on decimals being unsupported, so I
  switched it to Utf8.
• No existing sqllogictest or TPC-H/TPC-DS plan outputs changed.

Reproducer from the issue (TPC-H SF1, current main):

SELECT	Before	After	Actual
Date range	869,535	869,535	909,455
Date + Decimal conditions	1,200,243	111,291	114,160

Worth a closer look in review:

• The cast change extends the allowlist added in fix: restrict reverse cast constraint propagation to safe conversions #25531. Decimal widening is
  lossless and order-preserving, and casting bounds back rounds to a
  neighboring value, which only widens the range. That reasoning is what
  makes it safe, so I'd appreciate a second opinion on it.
• A decimal column with no min/max stats now gets selectivity 1.0 instead of
  the default, which matches how integer columns already behave.

Are there any user-facing changes?

Better row estimates for filters on decimal columns, which may change some
plan choices. One new public function, check_statistics_support; no breaking
changes.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 88.74172% with 34 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.94%. Comparing base (79650ec) to head (b887a68).
⚠️ Report is 20 commits behind head on main.

Files with missing lines	Patch %	Lines
datafusion/expr-common/src/interval_arithmetic.rs	88.42%	4 Missing and 10 partials ⚠️
datafusion/physical-plan/src/filter.rs	87.09%	4 Missing and 8 partials ⚠️
datafusion/physical-expr/src/expressions/cast.rs	82.60%	2 Missing and 6 partials ⚠️

Additional details and impacted files

@@           Coverage Diff            @@
##             main   #25790    +/-   ##
========================================
  Coverage   82.94%   82.94%            
========================================
  Files        1156     1156            
  Lines      458581   458871   +290     
  Branches   458581   458871   +290     
========================================
+ Hits       380382   380630   +248     
- Misses      55145    55162    +17     
- Partials    23054    23079    +25     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[kosiew]

hi @shrivasshankar
I would love to review this after the merge conflicts are resolved.

[shrivasshankar]

hi @shrivasshankar I would love to review this after the merge conflicts are resolved.

Thanks @kosiew! Conflicts are resolved now, ready for review.

[kosiew]

@shrivasshankar,

Thanks for working on decimal filter selectivity. The approach is appropriate, and the focused tests cover important decimal boundaries and cast behavior.

Keeping decimal support specific to statistics estimation, rather than expanding the existing symmetric hash join eligibility gate, is the right architectural boundary. The shared expression traversal also avoids unnecessary duplication. However, enabling decimal types must not admit arithmetic propagation that violates source column type invariants.

I'm requesting changes because valid decimal arithmetic predicates can now fail filter construction or produce column statistics with incorrect decimal types. These are correctness regressions that need to be addressed before merging. The additional SQL-level and cast-boundary tests are useful but non-blocking suggestions.

[kosiew]

Decimal arithmetic can propagate bounds with the wrong source type. For a: Decimal128(15,2) with statistics [1.00,10.00], a * Decimal128(2.00,15,2) > Decimal128(5.0000,31,4) makes FilterExec::try_new fail with Selectivity is out of limit: 8324028856.826859 because inverse propagation retains scale 4; a - 2.00 > 5.00 also publishes Decimal128(17,2) bounds for a Decimal128(15,2) column. Please restore source-typed bounds with conservative outward rounding, or exclude decimal arithmetic from statistics eligibility. Clamping selectivity alone is insufficient. Add regressions for multiplication construction/selectivity and subtraction min/max types.

[kosiew]

Optional follow-up: consider a SQLLogicTest for a Q6-style decimal range with ordinary fractional literals and known min/max statistics, asserting EXPLAIN estimates and query results. The current mixed-scale test constructs the cast explicitly, so it does not cover SQL coercion through physical expression construction to statistics estimation.

[kosiew]

Optional follow-up: consider representative containment tests for negative scales, cross-width widening, and bounds near precision limits. The exhaustive test covers only Decimal128 scale 2 to 3 near zero, while the Decimal32 to Decimal128 case currently checks eligibility rather than boundary propagation.

[shrivasshankar]

@shrivasshankar,

Thanks for working on decimal filter selectivity. The approach is appropriate, and the focused tests cover important decimal boundaries and cast behavior.

Keeping decimal support specific to statistics estimation, rather than expanding the existing symmetric hash join eligibility gate, is the right architectural boundary. The shared expression traversal also avoids unnecessary duplication. However, enabling decimal types must not admit arithmetic propagation that violates source column type invariants.

I'm requesting changes because valid decimal arithmetic predicates can now fail filter construction or produce column statistics with incorrect decimal types. These are correctness regressions that need to be addressed before merging. The additional SQL-level and cast-boundary tests are useful but non-blocking suggestions.

Thanks @kosiew, good catches!

I went with excluding decimal arithmetic from statistics estimation, so predicates like a * 2.00 > 5.0000 now fall back to the default selectivity. Comparisons, casts and negation still get estimates. Fixing the types through arithmetic seemed like a bigger change, so I'd leave that for a follow-up.

Added tests for your multiplication and subtraction cases, plus both of your suggestions: a sqllogictest with Q6-style fractional literals on a parquet table, and more cast boundary cases (cross-width, negative scale, near the precision limits).

[kosiew]

@shrivasshankar,

Thanks for the follow-up changes. The Decimal filter selectivity improvements now have appropriate correctness safeguards and regression coverage. The approach looks good.

Keeping Decimal arithmetic outside interval-based statistics estimation is the right architectural boundary for now. Arithmetic can change precision and scale, so propagating constraints without accounting for those type changes could produce incorrect column bounds. Retaining the default selectivity for unsupported expressions while allowing safe Decimal comparisons, widening casts, and negation is a sound approach.

The previous correctness blocker is addressed, and the additional tests cover the requested arithmetic fallback, mixed-scale coercion, and end-to-end SQL behavior. I found no remaining actionable issues. Approved.

[shrivasshankar]

@shrivasshankar,

Thanks for the follow-up changes. The Decimal filter selectivity improvements now have appropriate correctness safeguards and regression coverage. The approach looks good.

Keeping Decimal arithmetic outside interval-based statistics estimation is the right architectural boundary for now. Arithmetic can change precision and scale, so propagating constraints without accounting for those type changes could produce incorrect column bounds. Retaining the default selectivity for unsupported expressions while allowing safe Decimal comparisons, widening casts, and negation is a sound approach.

The previous correctness blocker is addressed, and the additional tests cover the requested arithmetic fallback, mixed-scale coercion, and end-to-end SQL behavior. I found no remaining actionable issues. Approved.

Thanks for the review @kosiew! Updated the branch with main.