fix: avoid pruning timestamp-to-time casts by rgbuilds · Pull Request #26452 · apache/datafusion

rgbuilds · 2026-10-17T18:59:28Z

Which issue does this PR close?

Closes Incorrect Parquet pruning for timestamp-to-time casts across midnight #26451.

Rationale for this change

Casting timestamp statistics to time-of-day values does not preserve ordering across midnight. This can invert the minimum and maximum used by statistics pruning and incorrectly discard files or row groups that contain matching rows.

What changes are included in this PR?

Prevent timestamp-to-Time32 and timestamp-to-Time64 casts from being used for statistics pruning.
Add pruning-expression coverage for both CAST and TRY_CAST across all supported time units.
Add a Parquet regression test whose timestamp bounds cross midnight.

What is the testing strategy for this PR?

The regression test verifies equality and range predicates for Time32 and Time64, including the expected row counts and pruning metrics. The full extended workspace test suite also passes.

Are there any user-facing changes?

Queries that cast timestamps to time-of-day values may perform less statistics pruning, but they no longer risk incorrect results when timestamp bounds cross a day boundary.

codecov-commenter · 2026-10-17T19:40:01Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.89%. Comparing base (484183d) to head (c874f76).

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #26452      +/-   ##
==========================================
- Coverage   82.89%   82.89%   -0.01%     
==========================================
  Files        1154     1154              
  Lines      457630   457665      +35     
  Branches   457630   457665      +35     
==========================================
+ Hits       379370   379394      +24     
- Misses      55256    55263       +7     
- Partials    23004    23008       +4

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Omega359 · 2026-10-21T19:31:37Z

I'm wondering if it's possible to have sqlogictests that would cover this? 🤔

Omega359 · 2026-10-21T22:08:17Z

    }

+    #[test]
+    fn row_group_predicate_cast_timestamp_to_time() {


Suggested change

fn row_group_predicate_cast_timestamp_to_time() {

fn row_group_predicate_cast_timestamp_to_time() {

// A timezone shifts where the day boundary falls but does not remove

// it, so a timezone-aware timestamp is just as unsafe to prune on.

for timestamp_type in [

DataType::Timestamp(TimeUnit::Second, None),

DataType::Timestamp(TimeUnit::Second, Some("+05:00".into())),

DataType::Timestamp(TimeUnit::Nanosecond, Some("UTC".into())),

] {

assert_timestamp_to_time_cast_is_not_prunable(timestamp_type);

}

}

/// Asserts that no comparison of `CAST`/`TRY_CAST` from `timestamp_type` to

/// a time-of-day produces a pruning expression.

fn assert_timestamp_to_time_cast_is_not_prunable(timestamp_type: DataType) {

I had claude update the test to handle timestamp with tz as I wanted to be sure that this fix covered that case as well.

Thanks — added this in 1fc4b25. The helper now runs the same CAST/TRY_CAST assertions across all Time32/Time64 units for timestamps without a timezone, with +05:00, and with UTC at nanosecond precision.

Omega359 · 2026-10-22T01:40:56Z

I got wondering just how difficult it would be to have tests for this as slt tests so I once again pointed claude at it and here is what it came up with. All 6 Parquet-backed queries in this slt fail on main and pass in this PR.

datafusion/sqllogictest/test_files/parquet_cast_pruning.slt (license header elided):

# Regression tests for casts that cannot be applied to min/max statistics when
# pruning Parquet files.
#
# Pruning must never change a query's answer. A cast whose output does not
# preserve the ordering of its input can invert the bounds derived from
# statistics, which makes pruning discard row groups that do contain matching
# rows. Every predicate below is therefore run twice: once against an
# in-memory table, which is never pruned and so gives the ground truth, and
# once against the same rows stored as Parquet. The two must agree.

# disable the listing cache so DataFusion picks up changes from COPY statements
statement ok
set datafusion.runtime.list_files_cache_limit = "0K";

##########
# Casting a timestamp to a time-of-day wraps at midnight
#
# https://github.com/apache/datafusion/issues/26451
##########

# Three timestamps at 23:00, 00:00 and 01:00, written as a single row group so
# that the row group's min/max bracket a midnight. Casting those bounds to
# time-of-day turns them into 23:00:00 and 01:00:00, which excludes every value
# that actually lies between them.
statement ok
CREATE TABLE source_data(ts TIMESTAMP) AS VALUES
  (TIMESTAMP '2024-03-10 23:00:00'),
  (TIMESTAMP '2024-03-11 00:00:00'),
  (TIMESTAMP '2024-03-11 01:00:00');

statement ok
COPY (SELECT ts FROM source_data)
TO 'test_files/scratch/parquet_cast_pruning/timestamps.parquet'
STORED AS PARQUET
OPTIONS ('format.max_row_group_size' '3');

statement ok
CREATE EXTERNAL TABLE timestamps
STORED AS PARQUET
LOCATION 'test_files/scratch/parquet_cast_pruning/timestamps.parquet';

# Equality against the row at midnight.
query P
SELECT ts FROM source_data WHERE CAST(ts AS TIME) = TIME '00:00:00' ORDER B
----
2024-03-11T00:00:00

query P
SELECT ts FROM timestamps WHERE CAST(ts AS TIME) = TIME '00:00:00' ORDER BY ts;
----
2024-03-11T00:00:00

# Range predicate spanning the wrap point.
query P
SELECT ts FROM source_data WHERE CAST(ts AS TIME) < TIME '02:00:00' ORDER BY ts;
----
2024-03-11T00:00:00
2024-03-11T01:00:00

query P
SELECT ts FROM timestamps WHERE CAST(ts AS TIME) < TIME '02:00:00' ORDER BY ts;
----
2024-03-11T00:00:00
2024-03-11T01:00:00

# The same predicates spelled with arrow_cast, which also reaches Time32.
query P
SELECT ts FROM source_data
WHERE arrow_cast(ts, 'Time32(Second)') = arrow_cast('00:00:00', 'Time32(Second)')
ORDER BY ts;
----
2024-03-11T00:00:00

query P
SELECT ts FROM timestamps
WHERE arrow_cast(ts, 'Time32(Second)') = arrow_cast('00:00:00', 'Time32(Sec
ORDER BY ts;
----
2024-03-11T00:00:00

query P
SELECT ts FROM source_data
WHERE arrow_cast(ts, 'Time32(Second)') < arrow_cast('02:00:00', 'Time32(Second)')
ORDER BY ts;
----
2024-03-11T00:00:00
2024-03-11T01:00:00

query P
SELECT ts FROM timestamps
WHERE arrow_cast(ts, 'Time32(Second)') < arrow_cast('02:00:00', 'Time32(Second)')
ORDER BY ts;
----
2024-03-11T00:00:00
2024-03-11T01:00:00

statement ok
DROP TABLE timestamps;

##########
# The same wrap happens at the local midnight of a timezone-aware column
##########

# The same three local wall-clock times, tagged +05:00, so the instants
# actually stored are 18:00, 19:00 and 20:00 UTC on 2024-03-10. Those are
# monotonic and fall inside a single UTC day, so the min/max bounds only
# invert if the cast honours the column's timezone.
statement ok
CREATE TABLE source_data_tz AS
SELECT arrow_cast(ts, 'Timestamp(Nanosecond, Some("+05:00"))') AS ts FROM s

statement ok
DROP TABLE source_data;

statement ok
COPY (SELECT ts FROM source_data_tz)
TO 'test_files/scratch/parquet_cast_pruning/timestamps_tz.parquet'
STORED AS PARQUET
OPTIONS ('format.max_row_group_size' '3');

statement ok
CREATE EXTERNAL TABLE timestamps_tz
STORED AS PARQUET
LOCATION 'test_files/scratch/parquet_cast_pruning/timestamps_tz.parquet';

# Pin the shape of the data this section depends on: the local times have t
# straddle a local midnight while the stored instants stay monotonic within a
# single UTC day. Without this, a change to how a timezone is attached woul
# leave the queries below passing while no longer exercising the wrap at all.
query TIT
SELECT
  ts::string AS local_time,
  to_unixtime(ts) AS stored_instant,
  CAST(ts AS TIME)::string AS time_of_day
FROM timestamps_tz ORDER BY ts;
----
2024-03-10T23:00:00+05:00 1710093600 23:00:00
2024-03-11T00:00:00+05:00 1710097200 00:00:00
2024-03-11T01:00:00+05:00 1710100800 01:00:00

query P
SELECT ts FROM source_data_tz WHERE CAST(ts AS TIME) = TIME '00:00:00' ORDE
----
2024-03-11T00:00:00+05:00

query P
SELECT ts FROM timestamps_tz WHERE CAST(ts AS TIME) = TIME '00:00:00' ORDER BY ts;
----
2024-03-11T00:00:00+05:00

query P
SELECT ts FROM source_data_tz WHERE CAST(ts AS TIME) < TIME '02:00:00' ORDE
----
2024-03-11T00:00:00+05:00
2024-03-11T01:00:00+05:00

query P
SELECT ts FROM timestamps_tz WHERE CAST(ts AS TIME) < TIME '02:00:00' ORDER
----
2024-03-11T00:00:00+05:00
2024-03-11T01:00:00+05:00

statement ok
DROP TABLE timestamps_tz;

statement ok
DROP TABLE source_data_tz;

# Config reset
statement ok
RESET datafusion.runtime.list_files_cache_limit;

statement ok
RESET datafusion.catalog.create_default_catalog_and_schema;

rgbuilds · 2026-10-22T08:45:31Z

I got wondering just how difficult it would be to have tests for this as slt tests so I once again pointed claude at it and here is what it came up with. All 6 Parquet-backed queries in this slt fail on main and pass in this PR.

@Omega359 Thanks for putting this together!

I added parquet_cast_pruning.slt in 1fc4b25, including the in-memory versus Parquet comparisons for equality and range predicates, explicit arrow_cast coverage, and the +05:00 timezone case with the data-shape assertion.
I kept the Rust Parquet integration test as complementary coverage because it checks the pruning metrics directly and also covers Time64. The focused tests and the full workspace suite pass, including all 539 SLT files.

github-actions Bot added the core Core DataFusion crate label Oct 17, 2026

Omega359 reviewed Oct 21, 2026

View reviewed changes

fix: avoid pruning timestamp-to-time casts

d942bdf

rgbuilds force-pushed the issue-26451-timestamp-time-pruning branch from c874f76 to 1fc4b25 Compare October 22, 2026 08:32

github-actions Bot added the sqllogictest SQL Logic Tests (.slt) label Oct 22, 2026

test: expand timestamp-to-time pruning coverage

514e0e3

rgbuilds force-pushed the issue-26451-timestamp-time-pruning branch from 1fc4b25 to 514e0e3 Compare October 22, 2026 08:39

Navigation Menu

Repository navigation

fix: avoid pruning timestamp-to-time casts - #26452

fix: avoid pruning timestamp-to-time casts#26452
rgbuilds wants to merge 2 commits into
apache:mainfrom
rgbuilds:issue-26451-timestamp-time-pruning

rgbuilds commented Oct 17, 2026

Uh oh!

codecov-commenter commented Oct 17, 2026

Uh oh!

Omega359 commented Oct 21, 2026

Uh oh!

Omega359 Oct 21, 2026

Uh oh!

rgbuilds Oct 22, 2026 •

edited

Loading

Uh oh!

Omega359 commented Oct 22, 2026

Uh oh!

rgbuilds commented Oct 22, 2026 •

edited

Loading

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

-    fn row_group_predicate_cast_timestamp_to_time() {
+    fn row_group_predicate_cast_timestamp_to_time() {
+        // A timezone shifts where the day boundary falls but does not remove
+        // it, so a timezone-aware timestamp is just as unsafe to prune on.
+        for timestamp_type in [
+            DataType::Timestamp(TimeUnit::Second, None),
+            DataType::Timestamp(TimeUnit::Second, Some("+05:00".into())),
+            DataType::Timestamp(TimeUnit::Nanosecond, Some("UTC".into())),
+        ] {
+            assert_timestamp_to_time_cast_is_not_prunable(timestamp_type);
+        }
+    }
+    /// Asserts that no comparison of `CAST`/`TRY_CAST` from `timestamp_type` to
+    /// a time-of-day produces a pruning expression.
+    fn assert_timestamp_to_time_cast_is_not_prunable(timestamp_type: DataType) {

Conversation

rgbuilds commented Oct 17, 2026

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

What is the testing strategy for this PR?

Are there any user-facing changes?

Uh oh!

codecov-commenter commented Oct 17, 2026

Codecov Report

Uh oh!

Omega359 commented Oct 21, 2026

Uh oh!

Omega359 Oct 21, 2026

Choose a reason for hiding this comment

Uh oh!

rgbuilds Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Omega359 commented Oct 22, 2026

Uh oh!

rgbuilds commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

rgbuilds Oct 22, 2026 •

edited

Loading

rgbuilds commented Oct 22, 2026 •

edited

Loading