docs: document timestamp and time zone semantics

[adriangb]

Which issue does this PR close?

• Closes Document timezone semantics #10344.

This is the last unchecked Documentation item in the EPIC
#8282.

Rationale for this change

DataFusion has a long tail of open timezone bugs:
#13212,
#12218,
#12892,
#23841,
#25095,
#18228,
#13962.

They look like seven unrelated bugs. They are the same confusion showing up at
seven different layers: when a timestamp gains or loses a time zone, is that a
relabel (keep the integer, change the annotation) or a shift (keep the
instant, change the wall clock)? Arrow's cast kernel says shift. Parts of
DataFusion — type coercion, unwrap_cast_in_comparison, to_timestamp_*,
::timestamptz when datafusion.execution.time_zone is unset — assume relabel.

That is already user-visible as a straight contradiction. On today's main,
with no session timezone set at all:

CREATE OR REPLACE TABLE c AS SELECT TIMESTAMP '2024-01-01 12:00:00' AS ts;

SELECT TIMESTAMP '2024-01-01 12:00:00'
       = arrow_cast(1704135600, 'Timestamp(Second, Some("America/Denver"))') AS literal_cmp;
-- true   (the naive side is shifted into Denver)

SELECT ts = arrow_cast(1704135600, 'Timestamp(Second, Some("America/Denver"))') AS column_cmp FROM c;
-- false  (the optimizer unwraps the cast and relabels)

Three PRs are in flight right now, each settling one piece of this
independently:

• fix: use session timezone for timestamp subtraction #25094 (@kumarUjjawal) — session
  timezone for timestamp subtraction / coercion
• fix: unwrap_cast_in_comparison drops the timezone shift when unwrapping CAST(timestamp AS timestamptz) = literal #25099 (@Ruchirtripathi) —
  unwrap_cast_in_comparison dropping the timezone shift
• fix: preserve the timezone in the schema when resolving output timezone #23925 (@nevi-me) — preserving the
  timezone in to_timestamp_*

With nothing written down, those three can each settle it differently, and we
end up with a fourth reading. This page is the shared reference that lets them
agree: it states the rule once, in one place, derived from what the code
actually does today.

@kumarUjjawal @Ruchirtripathi @nevi-me — could you each check your change
against the "The one rule" section of the new page and say whether it moves
DataFusion toward or away from it? If any of you thinks the rule as written is
the wrong rule, that is exactly the discussion this PR is trying to start, and
it is much cheaper to have it here than after three PRs land.

What changes are included in this PR?

A new user-guide page, docs/source/user-guide/sql/timestamps.md
("Timestamps and Time Zones"), added to the SQL Reference toctree immediately
after data_types.md.

It is a new page rather than a section of data_types.md because the content is
cross-cutting: it spans the type mapping, cast semantics, a session config
setting, the date/time function catalogue, DST, and query recipes.
data_types.md is a terse type-mapping reference and would be swamped by it;
operators.md is the wrong home for a data-model discussion.

Sections:

1. The data model — Timestamp(unit, Some(tz)) is an instant plus a display
   annotation; Timestamp(unit, None) is a wall clock with no instant. How that
   maps to SQL TIMESTAMP / TIMESTAMP WITH TIME ZONE, and where it does not.
2. The one rule — naive → zoned shifts; zoned → zoned
   relabels; zoned → naive yields the UTC wall clock. Each with a
   runnable example. Then AT TIME ZONE expressed in the same terms, and a
   subsection showing where DataFusion does not apply the rule consistently
   today.
3. The session time zone — what datafusion.execution.time_zone does and
   does not affect, with the default-unset footgun called out explicitly
   (including the exact shape of
   Grouping by utc timestamp turned into timezoned timestamp still uses utc for group #13962), and the type-unification
   rules for mixed-zone values.
4. Local time versus the instant — a table of which date/time functions work
   on the value's own local wall clock and which work on the UTC instant. The
   date_trunc / date_bin disagreement is spelled out; both individually match
   PostgreSQL, but the pair surprises everyone.
5. Daylight saving time — INTERVAL '1 day' vs INTERVAL '24 hours'
   (82800 / 86400 / 90000 seconds around the Denver transitions), and what
   currently happens on ambiguous and nonexistent local times.
6. Recipes — aggregating UTC data by local calendar day in a named zone
   (this produces exactly the output requested in
   Support date_bin on timestamps with timezone, properly accounting for Daylight Savings Time #10602), getting a zone's local
   wall clock, and round-tripping safely.
7. Differences from PostgreSQL — a table, each row linked to its issue by
   full URL.

Two smaller changes come with it:

• docs/source/user-guide/sql/data_types.md gains the missing
  TIMESTAMP WITH TIME ZONE row in the Date/Time table (it was simply absent),
  a note on precision, and a link to the new page.
• docs/source/user-guide/sql/index.rst gains the toctree entry.

What is the testing strategy for this PR?

Documentation only, so there is no code to test — but every claim on the page
was derived from behaviour rather than from reading the source, and every SQL
example is one that was actually run with its real output pasted in. Nothing on
the page is invented.

• All examples were run against datafusion-cli built from this branch
  (DataFusion 55.0.0, arrow-rs 59.2.0), including the error messages.
• All PostgreSQL claims were verified against a real PostgreSQL 17.11
  (docker run --rm -e POSTGRES_PASSWORD=pw postgres:17), not against the
  documentation: date_trunc / date_bin / date_part behaviour,
  INTERVAL '1 day' vs '24 hours' across both DST transitions,
  AT TIME ZONE on naive and aware inputs, tstz::timestamp, naive-vs-aware
  comparison, and DST-boundary resolution.
• The AT TIME ZONE-on-an-aware-value divergence was additionally confirmed
  against DuckDB.
• ./ci/scripts/doc_prettier_check.sh passes.
• The docs build (make html) was run and the new page renders; every
  cross-reference anchor it emits was checked to exist in the generated HTML.
  The build's only warning is the pre-existing missing deps.svg, which
  build.sh generates via cargo depgraph before make html.

A note on scope: the page documents today's behaviour, including behaviour
that is arguably wrong. Where an open issue proposes changing it, the page says
so and links the issue rather than documenting a future state. In particular the
AT TIME ZONE-on-an-aware-value rows carry an explicit note that
#12218 may change them.

Are there any user-facing changes?

Yes — new user documentation. No API or behaviour changes.

🤖 Generated with Claude Code

[adriangb]

Closing this for now.

On reflection this page documents several things that are bugs or unresolved inconsistencies, and writing them down as reference material presents them as designed and intended. That is the wrong signal — a table showing that date_bin and date_trunc disagree, or explaining how ::timestamptz erases a timezone, makes those behaviours look like semantics to learn rather than defects to fix.

The findings that motivated those sections have been filed as issues instead, so nothing is lost:

• TIMESTAMP WITH TIME ZONE can resolve to a timezone-naive type, and casting to it discards an existing timezone #25166 — TIMESTAMP WITH TIME ZONE resolving to a naive type, and casting to timestamptz discarding an existing timezone (the root cause of Grouping by utc timestamp turned into timezoned timestamp still uses utc for group #13962, includes the history of how the default changed in Change default time_zone to None (was "+00:00")  #18359)
• date_bin and date_trunc disagree on timezone-aware timestamps #25167 — date_bin and date_trunc disagreeing on timezone-aware values
• Document that date_bin with an explicit origin drifts an hour across a DST transition #25168 — date_bin with an explicit origin drifting across a DST transition
• generate_series over timestamps accepts only nanosecond precision, with a misleading error #25169 — generate_series accepting only nanosecond timestamps, with a misleading error
• AT TIME ZONE '+05:30' uses the opposite sign convention from PostgreSQL #25170 — AT TIME ZONE '+05:30' using the opposite sign convention from PostgreSQL

#10344 stays open. Documenting timezone semantics is still worth doing — but it should describe behaviour we intend to keep, which means it wants writing after the issues above are resolved or explicitly accepted, not before.

One piece of this is worth keeping regardless, and I will send it separately: data_types.md is currently missing the TIMESTAMP WITH TIME ZONE row altogether.