Skip to content

[fix](nereids) Prefer DATETIMEV2 over TIMESTAMPTZ when binding *_diff functions on non-literal string args - #67238

Open
lets-order-some-fries wants to merge 1 commit into
apache:masterfrom
lets-order-some-fries:fix-diff-varchar-tz-order
Open

[fix](nereids) Prefer DATETIMEV2 over TIMESTAMPTZ when binding *_diff functions on non-literal string args#67238
lets-order-some-fries wants to merge 1 commit into
apache:masterfrom
lets-order-some-fries:fix-diff-varchar-tz-order

Conversation

@lets-order-some-fries

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: close #66120

Related PR: #64127

Problem Summary:

DATEDIFF and the other eleven *_diff scalar functions silently return off-by-one (or off-by-N) results when (1) an argument is a string-typed non-literal — a varchar column, subquery projection slot, or UNION-ALL output slot — and (2) the session time_zone is not UTC.

All 12 functions list their TIMESTAMPTZ signature first in SIGNATURES. The timezone-coercion penalty in SearchSignature.doMatchTypes only fires for literals it can inspect (ExpressionUtils.getLiteralAfterUnwrapNullable, added in #64127), so for a varchar SlotReference every candidate signature ties and the tie-break keeps the first-listed candidate — TIMESTAMPTZ. The varchar value is then cast through timestamptz(6), which treats it as session-local time and shifts it to UTC, so day-number arithmetic runs on the shifted value.

Fix, as proposed by @Baymine in #66120: move the TimeStampTz signature to last in each of the 12 *Diff files. Literal-with-timezone arguments still route to TIMESTAMPTZ via the timeZoneCoersionScore branch; typed TIMESTAMPTZ / DATETIMEV2 / DATEV2 arguments still bind their identical-match signatures. Only the previously order-decided tie for non-literal string arguments changes: TIMESTAMPTZ → DATETIMEV2 (wall-clock semantics, independent of session time zone).

New unit test DiffFunctionSignatureTest covers all 12 functions in both directions: varchar slots must bind DATETIMEV2 (fails on master before this change) and TIMESTAMPTZ slots must still bind TIMESTAMPTZ. Happy to also add a SQL regression suite mirroring the issue's reproducer (varchar column / subquery slot / UNION-ALL slot under +08:00) if wanted — omitted here because the .out expected-files need a cluster run to generate.

Release note

Fixed DATEDIFF and the other *_diff functions returning wrong results on varchar columns and subquery slots when the session time_zone is not UTC.

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes. For *_diff calls whose string-typed argument is NOT a literal, the argument now binds to DATETIMEV2 (wall-clock) instead of TIMESTAMPTZ (UTC-shifting). That re-bind is the fix; literal arguments and typed date/datetime/timestamptz arguments are unaffected.
  • Does this need documentation?

    • No.
    • Yes.

… functions on non-literal string args

All 12 *_diff scalar functions listed their TIMESTAMPTZ signature first in
SIGNATURES. SearchSignature's timezone-coercion penalty only fires for literals
it can inspect (ExpressionUtils.getLiteralAfterUnwrapNullable, apache#64127), so for
a string-typed non-literal argument (varchar column, subquery projection slot,
UNION-ALL output slot) every candidate signature ties and the tie-break keeps
the first-listed candidate: TIMESTAMPTZ. The value is then cast through
timestamptz(6), which treats it as session-local time and shifts it to UTC, so
results are off-by-one under any non-UTC session time_zone.

Move the TimeStampTz signature to last in each file. Literal-with-timezone
arguments still route to TIMESTAMPTZ via the timeZoneCoersionScore branch, and
typed TIMESTAMPTZ / DATETIMEV2 / DATEV2 arguments still bind their
identical-match signatures; only the previously order-decided tie for
non-literal string arguments changes, to DATETIMEV2 (wall-clock semantics).

New DiffFunctionSignatureTest covers all 12 functions in both directions:
varchar slots bind DATETIMEV2 (fails before this change), timestamptz slots
still bind TIMESTAMPTZ.

Fixes apache#66120
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug](nereids) DATEDIFF and all *_diff functions silently produce off-by-one results on varchar columns / subquery slots when session time_zone != UTC

2 participants