Skip to content

Support timestamp cast literals in table-model time planning - #19223

Open
shuwenwei wants to merge 5 commits into
apache:masterfrom
shuwenwei:fix/prepared-timestamp-generic-literal
Open

shuwenwei wants to merge 5 commits into
apache:masterfrom
shuwenwei:fix/prepared-timestamp-generic-literal

Conversation

@shuwenwei

@shuwenwei shuwenwei commented Nov 11, 2026 •

Copy link
Copy Markdown
Member

Summary

  • Accept GenericLiteral timestamp bounds when converting table-model time filters.
  • Recognize the same timestamp literals during global time-predicate extraction so DeviceTableScanNode retains its TimePredicate and partition pruning remains effective.
  • Evaluate constant CAST(... AS TIMESTAMP) expressions before gap-fill range inference, including comparison and BETWEEN bounds.
  • Add integration coverage for comparison, IN, BETWEEN, EXPLAIN time-predicate extraction, and gap-fill range inference.

Reproduction Data

CREATE DATABASE pr_timestamp_planning;
USE pr_timestamp_planning;

CREATE TABLE employees(
  department_id STRING TAG,
  name TEXT FIELD
);

INSERT INTO employees(time, department_id, name)
VALUES(1, 'D001', 'Mary');
INSERT INTO employees(time, department_id, name)
VALUES(2, 'D001', 'John');
INSERT INTO employees(time, department_id, name)
VALUES(3, 'D002', 'Nancy');
INSERT INTO employees(time, department_id, name)
VALUES(4, 'D002', 'Jack');
INSERT INTO employees(time, department_id, name)
VALUES(5, 'D003', 'Linda');

CREATE TABLE table1(
  city STRING TAG,
  device_id STRING TAG,
  s1 DOUBLE FIELD
);

INSERT INTO table1(time, city, device_id, s1)
VALUES(2024-09-24T07:16:15.297+00:00, 'shanghai', 'd1', 27.2);
INSERT INTO table1(time, city, device_id, s1)
VALUES(2024-09-24T08:16:21.907+00:00, 'shanghai', 'd1', 27.3);
INSERT INTO table1(time, city, device_id, s1)
VALUES(2024-09-24T11:16:28.821+00:00, 'shanghai', 'd1', 29.3);

Problems And Fixes

1. Time filters using CAST(... AS TIMESTAMP)

SELECT time, name
FROM employees
WHERE time >= CAST(2 AS TIMESTAMP)
  AND time < CAST(5 AS TIMESTAMP)
ORDER BY time;

Constant folding represents CAST(2 AS TIMESTAMP) as a timestamp GenericLiteral, but ConvertPredicateToTimeFilterVisitor only accepted LongLiteral. The CAST bound could therefore reach the visitor as a GenericLiteral and fail with GenericLiteral cannot be cast to LongLiteral.

The same problem affected multi-value and single-value IN predicates:

SELECT time, name
FROM employees
WHERE time IN (CAST(2 AS TIMESTAMP), CAST(4 AS TIMESTAMP))
ORDER BY time;

SELECT time, name
FROM employees
WHERE time IN (CAST(2 AS TIMESTAMP))
ORDER BY time;

After the fix, timestamp GenericLiteral values are converted through the same long-value path as LongLiteral. The range query returns records at times 2, 3, and 4; the multi-value IN query returns records at times 2 and 4; and the single-value IN query returns the record at time 2.

2. Missing time predicates for partition pruning

EXPLAIN (FORMAT JSON)
SELECT time, name
FROM employees
WHERE time >= CAST(2 AS TIMESTAMP)
  AND time < CAST(5 AS TIMESTAMP)
ORDER BY time;

GlobalTimePredicateExtractVisitor previously recognized only LongLiteral timestamp bounds. Consequently, GenericLiteral bounds produced by timestamp CAST constant folding were not extracted as global time predicates. The table scan then had no TimePredicate, so partition pruning could be omitted and the query could schedule unnecessary partitions.

The same omission affected comparison and BETWEEN predicates. After the fix, the generated plan contains DeviceTableScanNode.properties.TimePredicate for comparison, BETWEEN, and single-value IN forms.

3. Gap-fill queries using CAST timestamp boundaries

SELECT date_bin_gapfill(1h, time) AS hour_time,
       avg(s1) AS avg_s1
FROM table1
WHERE time >= CAST(1727150400000 AS TIMESTAMP)
  AND time < CAST(1727179200000 AS TIMESTAMP)
  AND device_id = 'd1'
GROUP BY 1
ORDER BY 1;

GapFillStartAndEndTimeExtractVisitor previously accepted only LongLiteral boundaries. A constant CAST(... AS TIMESTAMP) could not be used to infer the range, so the query failed with:

could not infer startTime or endTime from WHERE clause

The visitor now evaluates symbol-free constant expressions before range inference and only falls back to an empty range when the expression is non-constant. Internal conversion errors are not broadly suppressed: symbol-containing expressions are checked first, and only VerifyException from non-constant verification is handled.

The same handling is applied to BETWEEN. The integration test uses < endTime and BETWEEN startTime AND endTime - 1 so both queries cover the same half-open time range.

Validation

  • mvn spotless:apply -pl iotdb-core/datanode
  • mvn spotless:apply -f integration-test/pom.xml
  • git diff --check
  • No compile or tests were run locally.

Accept GenericLiteral timestamp bounds when converting table-model time predicates, and cover comparison and IN filters with integration tests.
Recognize timestamp GenericLiteral bounds in comparison and BETWEEN predicates, and verify the extracted time predicate through EXPLAIN in integration tests.
Evaluate constant timestamp expressions when inferring gap-fill ranges, including CAST bounds, and cover comparison and BETWEEN forms in integration tests.
@shuwenwei shuwenwei changed the title Fix table-model time filters with timestamp cast literals Support timestamp cast literals in table-model time planning Nov 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants