From e2b397e5435353d4a37f3c7887633950ad31804f Mon Sep 17 00:00:00 2001 From: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com> Date: Thu, 10 Sep 2026 12:19:34 -0500 Subject: [PATCH 1/4] docs: document timestamp and time zone semantics Adds `docs/source/user-guide/sql/timestamps.md`, a new SQL Reference page that states DataFusion's timestamp/time zone model in one place, and links it from `data_types.md` and the SQL Reference toctree. The page covers: - the Arrow data model (`Timestamp(unit, Some(tz))` is an instant plus a display annotation; `Timestamp(unit, None)` is a wall clock with no instant) and how it maps to SQL's `TIMESTAMP` / `TIMESTAMP WITH TIME ZONE` - the one rule for conversions: naive -> zoned shifts, zoned -> zoned relabels, zoned -> naive yields the UTC wall clock - `AT TIME ZONE` expressed in terms of that rule - `datafusion.execution.time_zone`: what it does and does not affect, and the default-unset footgun that makes `::timestamptz` a naive type - a table of which date/time functions operate on the local wall clock and which on the UTC instant - daylight saving time, including the `INTERVAL '1 day'` vs `INTERVAL '24 hours'` distinction and the DST-boundary cast errors - recipes, including aggregating UTC data by local calendar day - known divergences from PostgreSQL, each linked to its issue Every SQL example was run against `datafusion-cli` on this commit and the pasted output is the real output. PostgreSQL claims were checked against PostgreSQL 17.11 and the `AT TIME ZONE` divergence against DuckDB. Co-Authored-By: Claude Opus 5 --- docs/source/user-guide/sql/data_types.md | 21 +- docs/source/user-guide/sql/index.rst | 1 + docs/source/user-guide/sql/timestamps.md | 687 +++++++++++++++++++++++ 3 files changed, 703 insertions(+), 6 deletions(-) create mode 100644 docs/source/user-guide/sql/timestamps.md diff --git a/docs/source/user-guide/sql/data_types.md b/docs/source/user-guide/sql/data_types.md index 2e75fce6c7aa9..bff2f4bb19c4e 100644 --- a/docs/source/user-guide/sql/data_types.md +++ b/docs/source/user-guide/sql/data_types.md @@ -105,12 +105,21 @@ The maximum supported precision for `DECIMAL` types is 76. ## Date/Time Types -| SQL DataType | Arrow DataType | -| ------------ | :------------------------------- | -| `DATE` | `Date32` | -| `TIME` | `Time64(Nanosecond)` | -| `TIMESTAMP` | `Timestamp(Nanosecond, None)` | -| `INTERVAL` | `Interval(IntervalMonthDayNano)` | +| SQL DataType | Arrow DataType | +| ------------------------------------------- | :------------------------------------------------------------ | +| `DATE` | `Date32` | +| `TIME` | `Time64(Nanosecond)` | +| `TIMESTAMP` | `Timestamp(Nanosecond, None)` | +| `TIMESTAMP WITH TIME ZONE` or `TIMESTAMPTZ` | `Timestamp(Nanosecond, Some(datafusion.execution.time_zone))` | +| `INTERVAL` | `Interval(IntervalMonthDayNano)` | + +`TIMESTAMP(p)` and `TIMESTAMP(p) WITH TIME ZONE` accept a precision `p` of 0, 3, +6 or 9, selecting second, millisecond, microsecond or nanosecond precision. + +Note that `datafusion.execution.time_zone` defaults to unset, in which case +`TIMESTAMP WITH TIME ZONE` maps to the timezone-**naive** +`Timestamp(Nanosecond, None)`. See [Timestamps and Time Zones](timestamps.md) +for what that means for casts, comparisons and the date/time functions. ## Boolean Types diff --git a/docs/source/user-guide/sql/index.rst b/docs/source/user-guide/sql/index.rst index f1fef45f705a8..35c8bc7e50aab 100644 --- a/docs/source/user-guide/sql/index.rst +++ b/docs/source/user-guide/sql/index.rst @@ -22,6 +22,7 @@ SQL Reference :maxdepth: 2 data_types + timestamps struct_coercion select subqueries diff --git a/docs/source/user-guide/sql/timestamps.md b/docs/source/user-guide/sql/timestamps.md new file mode 100644 index 0000000000000..12017f60553a2 --- /dev/null +++ b/docs/source/user-guide/sql/timestamps.md @@ -0,0 +1,687 @@ + + +# Timestamps and Time Zones + +Almost every surprising result involving timestamps in DataFusion comes down to a +single question: when a timestamp gains or loses a time zone, does the _instant_ +stay the same or does the _wall clock_ stay the same? This page answers that +question once, and then applies the answer to casts, the session time zone, the +date/time functions, daylight saving time, and a handful of recipes. + +All output on this page was produced with `datafusion-cli` on DataFusion 55.0.0 +with default settings unless a `SET` statement says otherwise. + +## The data model + +DataFusion timestamps are Arrow timestamps. Arrow has exactly two timestamp +shapes, and the difference between them is the whole story: + +| Arrow type | Physical value | Meaning | +| --------------------------- | ------------------------- | --------------------------------------------------------------------------------------------------------- | +| `Timestamp(unit, Some(tz))` | offset from the UTC epoch | An **instant**. `tz` is a display annotation: it says how to render the instant, not what the instant is. | +| `Timestamp(unit, None)` | a wall-clock reading | A **wall clock** with no instant attached. There is no fact about which point in time it names. | + +Two consequences follow, and they explain most of the rest of this page. + +- A zone-aware timestamp's `tz` **is not data**. Two values with the same + integer and different `tz` annotations are the _same instant_; comparing, + sorting, grouping or joining them treats them as equal. +- A zone-naive timestamp has **no instant**, so any operation that needs one has + to invent a zone. Which zone gets invented is the subject of + [The session time zone](#the-session-time-zone) below, and it is not always + the same zone. + +### How SQL types map to Arrow types + +| SQL type | Arrow type | +| ---------------------------------------------------------- | ------------------------------------------------------------- | +| `TIMESTAMP`, `TIMESTAMP WITHOUT TIME ZONE`, `::timestamp` | `Timestamp(Nanosecond, None)` | +| `TIMESTAMP WITH TIME ZONE`, `TIMESTAMPTZ`, `::timestamptz` | `Timestamp(Nanosecond, Some(datafusion.execution.time_zone))` | + +`TIMESTAMP(p)` with `p` of 0, 3, 6 or 9 selects second, millisecond, +microsecond or nanosecond precision respectively. + +The second row is the important one, and it is where DataFusion parts company +with PostgreSQL. `TIMESTAMP WITH TIME ZONE` resolves to whatever +`datafusion.execution.time_zone` is set to — and that setting **defaults to +unset**. With it unset, `TIMESTAMP WITH TIME ZONE` is a zone-_naive_ type: + +```sql +SELECT arrow_typeof('2024-01-01T12:00:00Z'::timestamptz) AS type, + '2024-01-01T12:00:00Z'::timestamptz AS value; +``` + +```text ++---------------+---------------------+ +| type | value | ++---------------+---------------------+ +| Timestamp(ns) | 2024-01-01T12:00:00 | ++---------------+---------------------+ +``` + +The `Z` was accepted and then discarded. PostgreSQL has no equivalent state: its +`TimeZone` parameter is always set to something, so `timestamptz` is always a +zone-aware type there. Set the session time zone and the same query behaves the +way a PostgreSQL user expects: + +```sql +SET datafusion.execution.time_zone = 'America/Denver'; + +SELECT arrow_typeof('2024-01-01T12:00:00Z'::timestamptz) AS type, + '2024-01-01T12:00:00Z'::timestamptz AS value; +``` + +```text ++---------------------------------+---------------------------+ +| type | value | ++---------------------------------+---------------------------+ +| Timestamp(ns, "America/Denver") | 2024-01-01T05:00:00-07:00 | ++---------------------------------+---------------------------+ +``` + +:::{note} +If you are writing SQL that must behave predictably, set +`datafusion.execution.time_zone` explicitly. `'UTC'` is a good default: +it makes `timestamptz` a genuinely zone-aware type, makes `now()` zone-aware, +and keeps every conversion on this page a no-op shift. +::: + +## The one rule + +Every conversion between the two shapes follows from one rule, applied in three +directions. + +### Zone-naive to zone-aware: a **shift** + +The naive wall clock is read as a local time _in the target zone_. The wall +clock is preserved; the instant changes. + +```sql +SELECT + TIMESTAMP '2024-01-01 12:00:00' AS naive, + arrow_cast(TIMESTAMP '2024-01-01 12:00:00', + 'Timestamp(Second, Some("America/Denver"))') AS zoned, + to_unixtime(TIMESTAMP '2024-01-01 12:00:00') AS naive_epoch, + to_unixtime(arrow_cast(TIMESTAMP '2024-01-01 12:00:00', + 'Timestamp(Second, Some("America/Denver"))')) AS zoned_epoch; +``` + +```text ++---------------------+---------------------------+-------------+-------------+ +| naive | zoned | naive_epoch | zoned_epoch | ++---------------------+---------------------------+-------------+-------------+ +| 2024-01-01T12:00:00 | 2024-01-01T12:00:00-07:00 | 1704110400 | 1704135600 | ++---------------------+---------------------------+-------------+-------------+ +``` + +The wall clock is still `12:00:00`; the epoch moved by 25200 seconds, the +`-07:00` offset. This is the direction that trips people up, because the +underlying integer changed even though nothing about the printed value did. + +### Zone-aware to zone-aware: a **relabel** + +The instant is preserved; only the display annotation changes. + +```sql +CREATE OR REPLACE VIEW utc AS + SELECT arrow_cast(TIMESTAMP '2024-01-01 12:00:00', + 'Timestamp(Second, Some("UTC"))') AS t; + +SELECT t AS in_utc, + arrow_cast(t, 'Timestamp(Second, Some("America/Denver"))') AS in_denver, + to_unixtime(t) AS utc_epoch, + to_unixtime(arrow_cast(t, 'Timestamp(Second, Some("America/Denver"))')) AS denver_epoch +FROM utc; +``` + +```text ++----------------------+---------------------------+------------+--------------+ +| in_utc | in_denver | utc_epoch | denver_epoch | ++----------------------+---------------------------+------------+--------------+ +| 2024-01-01T12:00:00Z | 2024-01-01T05:00:00-07:00 | 1704110400 | 1704110400 | ++----------------------+---------------------------+------------+--------------+ +``` + +### Zone-aware to zone-naive: the **UTC** wall clock + +The zone annotation is dropped and the integer is kept, which means the +resulting wall clock is the value's wall clock **in UTC** — regardless of what +the source annotation was and regardless of the session time zone. + +```sql +SELECT arrow_cast(t, 'Timestamp(Second, None)') AS zoned_to_naive FROM utc; + +SELECT arrow_cast(arrow_cast(t, 'Timestamp(Second, Some("America/Denver"))'), + 'Timestamp(Second, None)') AS denver_to_naive FROM utc; +``` + +```text ++---------------------+ +| zoned_to_naive | ++---------------------+ +| 2024-01-01T12:00:00 | ++---------------------+ + ++---------------------+ +| denver_to_naive | ++---------------------+ +| 2024-01-01T12:00:00 | ++---------------------+ +``` + +Both give `12:00:00`, the UTC wall clock, even though the second value displays +as `05:00:00-07:00`. PostgreSQL instead converts to the _session_ time zone +here; see [Differences from PostgreSQL](#differences-from-postgresql). + +To get a local wall clock instead, use +[`to_local_time`](scalar_functions.md#to_local_time) — see +[Recipes](#recipes). + +### Summary + +| Conversion | What is preserved | What changes | +| ------------------ | ----------------- | -------------------------- | +| naive → zoned | wall clock | the instant (shifted) | +| zoned → zoned | the instant | the display annotation | +| zoned → naive | the instant | becomes the UTC wall clock | + +### `AT TIME ZONE` + +`AT TIME ZONE` applies the same rule, chosen by the input's shape: + +- On a zone-**naive** input it is the naive → zoned shift: the wall clock + is read as local time in the named zone. +- On a zone-**aware** input it is the zoned → zoned relabel: the instant is + kept and the display annotation is replaced. + +```sql +CREATE OR REPLACE VIEW utc AS + SELECT arrow_cast(TIMESTAMP '2024-01-01 12:00:00', + 'Timestamp(Second, Some("UTC"))') AS t; + +SELECT arrow_typeof(TIMESTAMP '2024-01-01 12:00:00' AT TIME ZONE 'America/Denver') AS naive_input_type, + TIMESTAMP '2024-01-01 12:00:00' AT TIME ZONE 'America/Denver' AS naive_input; + +SELECT arrow_typeof(t AT TIME ZONE 'America/Denver') AS zoned_input_type, + t AT TIME ZONE 'America/Denver' AS zoned_input, + (t AT TIME ZONE 'America/Denver')::timestamp AS then_cast_to_naive +FROM utc; +``` + +```text ++---------------------------------+---------------------------+ +| naive_input_type | naive_input | ++---------------------------------+---------------------------+ +| Timestamp(ns, "America/Denver") | 2024-01-01T12:00:00-07:00 | ++---------------------------------+---------------------------+ + ++---------------------------------+---------------------------+---------------------+ +| zoned_input_type | zoned_input | then_cast_to_naive | ++---------------------------------+---------------------------+---------------------+ +| Timestamp(ns, "America/Denver") | 2024-01-01T05:00:00-07:00 | 2024-01-01T12:00:00 | ++---------------------------------+---------------------------+---------------------+ +``` + +The zone-naive case agrees with PostgreSQL: both name the instant +`2024-01-01T19:00:00Z`, DataFusion displaying it in Denver and PostgreSQL in the +session zone. + +The zone-aware case does not. PostgreSQL and DuckDB both return a zone-**naive** +`2024-01-01 05:00:00` there, so `(t AT TIME ZONE 'America/Denver')::timestamp` +gives `05:00:00` in those systems and `12:00:00` — the UTC wall clock — in +DataFusion. See ; this is +under active discussion and may change. + +### Where the rule is not applied consistently + +The rule above describes the cast kernel. Some parts of DataFusion take the +other reading — that a naive value is "really" UTC and gaining a zone is a +relabel — and that inconsistency is visible from SQL today. The clearest case: + +```sql +CREATE OR REPLACE TABLE c AS SELECT TIMESTAMP '2024-01-01 12:00:00' AS ts; + +-- 2024-01-01 12:00 in Denver is 2024-01-01T19:00:00Z (epoch 1704135600) +SELECT arrow_cast(ts, 'Timestamp(Second, Some("America/Denver"))') AS ts_in_denver FROM c; + +-- literal on both sides: the naive value is shifted, so this is true +SELECT TIMESTAMP '2024-01-01 12:00:00' + = arrow_cast(1704135600, 'Timestamp(Second, Some("America/Denver"))') AS literal_cmp; + +-- the same comparison with the naive value in a column: false +SELECT ts = arrow_cast(1704135600, 'Timestamp(Second, Some("America/Denver"))') AS column_cmp FROM c; +``` + +```text ++---------------------------+ +| ts_in_denver | ++---------------------------+ +| 2024-01-01T12:00:00-07:00 | ++---------------------------+ + ++-------------+ +| literal_cmp | ++-------------+ +| true | ++-------------+ + ++------------+ +| column_cmp | ++------------+ +| false | ++------------+ +``` + +The cast in the first query shifts. The comparison in the third query is +rewritten by the `unwrap_cast_in_comparison` optimizer rule, which removes the +cast and keeps the literal's integer unchanged — a relabel. Note that +`datafusion.execution.time_zone` is not set here and plays no part: the two +queries disagree purely because one path shifts and the other relabels. Until + is fixed, prefer to make the +conversion explicit and to compare zone-aware values against zone-aware values. + +## The session time zone + +`datafusion.execution.time_zone` (see [Configuration Settings](../configs.md)) +is the zone DataFusion uses when it has to invent one. It **defaults to unset**. + +What it affects: + +- The Arrow type that `TIMESTAMP WITH TIME ZONE` / `::timestamptz` resolves to. +- The type and value returned by `now()`, `current_timestamp`, `current_date` + and `current_time`. +- The zone that `to_timestamp` and the `to_timestamp_*` family produce, and the + zone that a zone-less string argument to them is interpreted in. + +What it does **not** affect: + +- The meaning of a `Timestamp(unit, Some(tz))` value that already exists. A + column read from Parquet or created by `AT TIME ZONE` keeps its own zone. +- `from_unixtime`, which always produces a zone-naive UTC wall clock + (). +- `date_part` / `EXTRACT` on a zone-naive value, which reports that value's + stored wall clock and never reinterprets it + (). +- Zone-aware → zone-naive casts, which always produce the UTC wall clock. + +```sql +SET datafusion.execution.time_zone = 'America/Denver'; + +SELECT arrow_typeof(now()) AS now, + arrow_typeof(current_timestamp) AS current_timestamp, + arrow_typeof(to_timestamp('2024-01-01T12:00:00')) AS to_timestamp, + arrow_typeof(from_unixtime(1704110400)) AS from_unixtime; + +SELECT to_timestamp('2024-01-01T12:00:00') AS to_timestamp_naive_string, + from_unixtime(1704110400) AS from_unixtime_value; +``` + +```text ++---------------------------------+---------------------------------+---------------------------------+---------------+ +| now | current_timestamp | to_timestamp | from_unixtime | ++---------------------------------+---------------------------------+---------------------------------+---------------+ +| Timestamp(ns, "America/Denver") | Timestamp(ns, "America/Denver") | Timestamp(ns, "America/Denver") | Timestamp(s) | ++---------------------------------+---------------------------------+---------------------------------+---------------+ + ++---------------------------+---------------------+ +| to_timestamp_naive_string | from_unixtime_value | ++---------------------------+---------------------+ +| 2024-01-01T12:00:00-07:00 | 2024-01-01T12:00:00 | ++---------------------------+---------------------+ +``` + +Note that `to_timestamp` read the zone-less string `'2024-01-01T12:00:00'` as +`12:00` _local_ Denver time, while `from_unixtime` returned a naive UTC wall +clock for the same epoch. + +### The default-unset footgun + +With `datafusion.execution.time_zone` unset, `::timestamptz` **erases** an +existing zone rather than converting to one, because the target type is +`Timestamp(_, None)` and the conversion is the zone-aware → zone-naive +rule: keep the integer, drop the annotation. Wrapping a correct expression in +`::timestamptz` therefore silently changes its meaning to UTC: + +```sql +CREATE OR REPLACE TABLE hits(t TIMESTAMP) AS VALUES + (TIMESTAMP '2024-04-30T21:30:00'), (TIMESTAMP '2024-04-30T22:30:00'), + (TIMESTAMP '2024-04-30T23:30:00'), (TIMESTAMP '2024-05-01T00:00:00'), + (TIMESTAMP '2024-05-01T00:30:00'), (TIMESTAMP '2024-05-01T10:30:00'), + (TIMESTAMP '2024-05-01T20:30:00'); +CREATE OR REPLACE VIEW hits_utc AS SELECT t AT TIME ZONE 'UTC' AS t FROM hits; + +-- correct: group on the zone-aware value +SELECT date_trunc('day', t AT TIME ZONE 'Europe/Brussels') AS day, count(*) AS n +FROM hits_utc GROUP BY 1 ORDER BY 1; + +-- wrong: ::timestamptz with datafusion.execution.time_zone unset +SELECT date_trunc('day', (t AT TIME ZONE 'Europe/Brussels')::timestamptz) AS day, count(*) AS n +FROM hits_utc GROUP BY 1 ORDER BY 1; +``` + +```text ++---------------------------+---+ +| day | n | ++---------------------------+---+ +| 2024-04-30T00:00:00+02:00 | 1 | +| 2024-05-01T00:00:00+02:00 | 6 | ++---------------------------+---+ + ++---------------------+---+ +| day | n | ++---------------------+---+ +| 2024-04-30T00:00:00 | 3 | +| 2024-05-01T00:00:00 | 4 | ++---------------------+---+ +``` + +The second query groups by UTC day even though it names Brussels. See +. + +### Combining values from different zones + +When two timestamps of different types meet — in a comparison, a `UNION`, a +`CASE`, a `coalesce` — DataFusion picks a common type: + +| Left | Right | Common type | +| ----------- | ------------- | ------------------------------------------ | +| `Some(tz)` | the same `tz` | that `tz` | +| `Some(tz1)` | `Some(tz2)` | `Some("UTC")` (both are relabelled) | +| `None` | `Some(tz)` | `Some(tz)` — the naive side is **shifted** | + +Because the common type of two _different_ named zones is UTC and not the +session zone, a `UNION` of Denver and Brussels data displays as UTC. Nothing is +lost: all three cases preserve every instant. + +## Local time versus the instant + +Some functions work on the value's **local wall clock** — the reading you get +after applying the value's own `tz` annotation. Others work on the **UTC +instant** and ignore the annotation except for display. Which group a function +falls into is rarely stated anywhere, and it is the difference between a correct +and an incorrect query, so here it is explicitly. + +For a zone-aware value, "local" means the value's _own_ zone, not the session +zone. For a zone-naive value there is no zone to apply: the local-wall-clock +functions use the stored reading as-is, and the instant-based functions treat it +as UTC. + +| Function / operator | Operates on | Notes | +| ------------------------------------------------------------------- | ------------------- | ---------------------------------------------------------------------------------- | +| [`date_trunc`](scalar_functions.md#date_trunc), `datetrunc` | local wall clock | Truncates to local midnight, local hour, … | +| [`date_part`](scalar_functions.md#date_part), `datepart`, `EXTRACT` | local wall clock | | +| [`to_char`](scalar_functions.md#to_char), `date_format` | local wall clock | | +| [`to_local_time`](scalar_functions.md#to_local_time) | local wall clock | Returns `Timestamp(unit, None)` holding the local reading | +| `CAST(t AS DATE)`, `CAST(t AS TIME)` | local wall clock | | +| `timestamp + interval`, `timestamp - interval` | local wall clock | Calendar units are DST-aware; see below | +| [`generate_series`](scalar_functions.md#generate_series) / `range` | local wall clock | Steps by calendar units in the value's own zone; accepts nanosecond precision only | +| [`date_bin`](scalar_functions.md#date_bin) | **the UTC instant** | Bins are anchored at the UTC epoch, not at local midnight | +| [`to_unixtime`](scalar_functions.md#to_unixtime) | **the UTC instant** | | +| `AT TIME ZONE`, `CAST(t AS TIMESTAMPTZ)` | **the UTC instant** | For a zone-aware input; a zone-naive input is shifted, see above | +| Comparison, `ORDER BY`, `GROUP BY`, joins, `min`/`max` | **the UTC instant** | Annotations are ignored; equal instants are equal | + +The consequence worth memorising is that `date_trunc` and `date_bin` disagree on +the same value: + +```sql +CREATE OR REPLACE VIEW v AS + SELECT arrow_cast(TIMESTAMP '2024-01-01 12:00:00', + 'Timestamp(Second, Some("America/Denver"))') AS t; + +SELECT t, + date_trunc('day', t) AS date_trunc, + date_bin(INTERVAL '1 day', t) AS date_bin +FROM v; +``` + +```text ++---------------------------+---------------------------+---------------------------+ +| t | date_trunc | date_bin | ++---------------------------+---------------------------+---------------------------+ +| 2024-01-01T12:00:00-07:00 | 2024-01-01T00:00:00-07:00 | 2023-12-31T17:00:00-07:00 | ++---------------------------+---------------------------+---------------------------+ +``` + +`date_trunc` gave local midnight in Denver. `date_bin` gave the start of the UTC +day, displayed in Denver (`2023-12-31T17:00:00-07:00` is `2024-01-01T00:00:00Z`). +Each individually matches PostgreSQL; the pair is still surprising. Use +`date_trunc` when you want local calendar boundaries, and see +[Recipes](#recipes) for local-calendar binning with `date_bin`. + +The other functions on a zone-aware value: + +```sql +SELECT date_part('hour', t) AS date_part_hour, + to_char(t, '%H:%M') AS to_char, + to_local_time(t) AS to_local_time, + t::date AS cast_to_date +FROM v; +``` + +```text ++----------------+---------+---------------------+--------------+ +| date_part_hour | to_char | to_local_time | cast_to_date | ++----------------+---------+---------------------+--------------+ +| 12 | 12:00 | 2024-01-01T12:00:00 | 2024-01-01 | ++----------------+---------+---------------------+--------------+ +``` + +## Daylight saving time + +### `INTERVAL '1 day'` is not `INTERVAL '24 hours'` + +This is the single most useful thing to know on this page. On a **zone-aware** +timestamp, DataFusion adds calendar units (years, months, days) in the value's +own local calendar and adds sub-day units (hours, minutes, seconds) as elapsed +time. Across a DST transition the two differ: + +```sql +CREATE OR REPLACE VIEW d AS + SELECT arrow_cast(TIMESTAMP '2024-03-09 12:00:00', + 'Timestamp(Second, Some("America/Denver"))') AS t; + +SELECT t, + t + INTERVAL '1 day' AS plus_1_day, + t + INTERVAL '24 hours' AS plus_24_hours, + to_unixtime(t + INTERVAL '1 day') - to_unixtime(t) AS seconds_1_day, + to_unixtime(t + INTERVAL '24 hours') - to_unixtime(t) AS seconds_24_hours +FROM d; +``` + +```text ++---------------------------+---------------------------+---------------------------+---------------+------------------+ +| t | plus_1_day | plus_24_hours | seconds_1_day | seconds_24_hours | ++---------------------------+---------------------------+---------------------------+---------------+------------------+ +| 2024-03-09T12:00:00-07:00 | 2024-03-10T12:00:00-06:00 | 2024-03-10T13:00:00-06:00 | 82800 | 86400 | ++---------------------------+---------------------------+---------------------------+---------------+------------------+ +``` + +America/Denver springs forward on 2024-03-10, so that local day is 23 hours +long. `INTERVAL '1 day'` lands on the same wall clock the next day (82800 +seconds later); `INTERVAL '24 hours'` lands 24 hours later, an hour further on +the clock. In the autumn the same pair goes the other way: adding +`INTERVAL '1 day'` to `2024-11-02 12:00:00-06:00` advances the instant by 90000 +seconds. + +This matches PostgreSQL exactly (verified against PostgreSQL 17). On a +**zone-naive** timestamp there is no DST to apply, so `INTERVAL '1 day'` and +`INTERVAL '24 hours'` always agree. + +Use calendar units when you mean "the same time tomorrow" and sub-day units when +you mean "24 hours of elapsed time". They are not interchangeable. + +### Ambiguous and nonexistent local times + +Two local wall clocks per year are not well defined in a zone with DST: the hour +skipped when the clocks go forward does not exist, and the hour repeated when +they go back is ambiguous. DataFusion currently **errors** on both, in both the +literal and the column path: + +```sql +SET datafusion.execution.time_zone = 'America/Denver'; +CREATE OR REPLACE TABLE gap AS SELECT TIMESTAMP '2024-03-10 02:30:00' AS ts; +SELECT ts::timestamptz FROM gap; +``` + +```text +Arrow error: Cast error: Cannot cast timezone to different timezone +``` + +```sql +SET datafusion.execution.time_zone = 'America/Denver'; +SELECT '2024-03-10T02:30:00'::timestamptz; +``` + +```text +Optimizer rule 'simplify_expressions' failed +caused by +Arrow error: Parser error: Error parsing timestamp from '2024-03-10T02:30:00': error computing timezone offset +``` + +The same happens for the ambiguous fall-back hour, `2024-11-03 01:30:00` in +America/Denver. PostgreSQL resolves both instead of erroring (the gap moves +forward, the ambiguous hour is read as standard time). + +Fixed-offset zones such as `'+08:00'` never have transitions and are never +affected. See and the +upstream fix . + +## Recipes + +### Aggregate UTC data by local calendar day in a named zone + +`date_bin` bins on the UTC instant, so binning zone-aware data directly gives +UTC days. Convert to the target zone, then flatten to a local wall clock with +`to_local_time`, and bin that: + +```sql +CREATE OR REPLACE TABLE hits(t TIMESTAMP) AS VALUES + (TIMESTAMP '2024-04-30T21:30:00'), (TIMESTAMP '2024-04-30T22:30:00'), + (TIMESTAMP '2024-04-30T23:30:00'), (TIMESTAMP '2024-05-01T00:00:00'), + (TIMESTAMP '2024-05-01T00:30:00'), (TIMESTAMP '2024-05-01T10:30:00'), + (TIMESTAMP '2024-05-01T20:30:00'); + +CREATE OR REPLACE VIEW hits_utc AS SELECT t AT TIME ZONE 'UTC' AS t FROM hits; + +SELECT t AT TIME ZONE 'Europe/Brussels' AS local, + date_bin(INTERVAL '1 day', + to_local_time(t AT TIME ZONE 'Europe/Brussels')) AS local_day +FROM hits_utc; +``` + +```text ++---------------------------+---------------------+ +| local | local_day | ++---------------------------+---------------------+ +| 2024-04-30T23:30:00+02:00 | 2024-04-30T00:00:00 | +| 2024-05-01T00:30:00+02:00 | 2024-05-01T00:00:00 | +| 2024-05-01T01:30:00+02:00 | 2024-05-01T00:00:00 | +| 2024-05-01T02:00:00+02:00 | 2024-05-01T00:00:00 | +| 2024-05-01T02:30:00+02:00 | 2024-05-01T00:00:00 | +| 2024-05-01T12:30:00+02:00 | 2024-05-01T00:00:00 | +| 2024-05-01T22:30:00+02:00 | 2024-05-01T00:00:00 | ++---------------------------+---------------------+ +``` + +`local_day` is a zone-naive value: it is a local calendar day label, not an +instant, which is exactly what you want as a `GROUP BY` key. For whole calendar +units, `date_trunc('day', t AT TIME ZONE 'Europe/Brussels')` gives the same +grouping while keeping the result zone-aware: + +```sql +SELECT date_trunc('day', t AT TIME ZONE 'Europe/Brussels') AS day, count(*) AS n +FROM hits_utc GROUP BY 1 ORDER BY 1; +``` + +```text ++---------------------------+---+ +| day | n | ++---------------------------+---+ +| 2024-04-30T00:00:00+02:00 | 1 | +| 2024-05-01T00:00:00+02:00 | 6 | ++---------------------------+---+ +``` + +Use the `to_local_time` form when the bin width is not a whole calendar unit +(`INTERVAL '15 minutes'`, `INTERVAL '4 hours'`), which `date_trunc` cannot +express. + +:::{warning} +Passing `date_bin` an origin in the target zone — for example +`date_bin(INTERVAL '1 day', t, TIMESTAMP '2024-04-30 22:00:00' AT TIME ZONE 'UTC')` +— appears to work but is only correct while the zone's offset does not change. +It silently drifts by an hour across a DST transition. Prefer `to_local_time`. +::: + +### Get a zone's local wall clock + +```sql +SELECT to_local_time(t AT TIME ZONE 'Europe/Brussels') AS brussels_wall_clock +FROM hits_utc LIMIT 1; +``` + +`AT TIME ZONE` relabels the instant for display in Brussels; `to_local_time` +then converts that display into an actual zone-naive value. Do **not** use +`::timestamp` for this — that returns the UTC wall clock, not the local one. + +### Round-trip safely + +- Store instants as `Timestamp(unit, Some("UTC"))` and convert to a display zone + only at the edge of the query. +- Set `datafusion.execution.time_zone = 'UTC'` so that `timestamptz`, `now()` + and `to_timestamp` are zone-aware and every conversion in this page is a + zero-offset shift. +- Use `arrow_cast` with an explicit Arrow type when you want the exact + conversion; `::timestamptz` depends on session configuration. +- Never round-trip a zone-aware value through `Timestamp(_, None)` unless you + intend to reduce it to a UTC wall clock. + +## Differences from PostgreSQL + +DataFusion aims to follow PostgreSQL, and does for most of the above. These are +the known divergences today. + +| Behaviour | DataFusion | PostgreSQL | Issue | +| -------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------- | --------------------------------------------------- | +| `TIMESTAMP WITH TIME ZONE` with no session zone set | Zone-**naive** `Timestamp(_, None)`; the offset in a literal is discarded | Always zone-aware; `TimeZone` is always set | — | +| `AT TIME ZONE` applied to a zone-**aware** value | Returns a zone-**aware** value in the named zone | Returns a zone-**naive** value | | +| `tstz::timestamp` | The **UTC** wall clock | The **session zone** wall clock | | +| Zone-naive value compared with a zone-aware one | The naive side is shifted into the **other operand's** zone, not the session zone | The naive side is read in the **session** zone | | +| Zone-naive **column** compared with a zone-aware literal | The optimizer drops the shift, giving the opposite answer to the same comparison written with literals | Consistent with the literal form | | +| `tstz - timestamp` | Naive side read as UTC, so the session offset is lost | Naive side read in the session zone | | +| Ambiguous / nonexistent local time | Error | Resolved (gap moves forward, ambiguity reads as standard time) | | +| `from_unixtime` | Zone-naive UTC wall clock; ignores the session zone | `to_timestamp(double)` returns `timestamptz` | | +| `date_part` / `EXTRACT` on a zone-naive value | Uses the stored wall clock; ignores the session zone | The same — but DataFusion's config docs once promised session-zone awareness | | +| `to_timestamp_*` on an already zone-aware input | Rewrites the zone to the session zone, dropping it entirely when unset | n/a | | + +:::{note} +The `AT TIME ZONE` row describes DataFusion's behaviour as of this writing. There +is active discussion in +about aligning the zone-**aware**-input case with PostgreSQL and DuckDB, which +would change the second and third rows above; DuckDB also returns a zone-naive +value there. The zone-**naive** input case already agrees with PostgreSQL: both +produce the same instant. +::: + +## See also + +- [Data Types](data_types.md) +- [Date and Time Functions](scalar_functions.md#time-and-date-functions) +- [Configuration Settings](../configs.md) From e7049e462afe112c0869e5ae8ebe8f89e5a9ec7a Mon Sep 17 00:00:00 2001 From: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com> Date: Thu, 10 Sep 2026 12:39:51 -0500 Subject: [PATCH 2/4] docs: rewrite timestamps page prose in Simplified Technical English Rewrite the prose of docs/source/user-guide/sql/timestamps.md to follow ASD-STE100 Issue 8: approved vocabulary, one term per concept ("aware"/"naive timestamp"), active voice, simple tenses, sentences of at most 25 words, instructions that start with the verb, and warnings that give the instruction before the explanation. No SQL block, result block, table data cell, URL, anchor target or technical fact changes. The 33 code fences are byte-identical to the previous commit. Co-Authored-By: Claude Opus 5 --- docs/source/user-guide/sql/timestamps.md | 491 +++++++++++++---------- 1 file changed, 272 insertions(+), 219 deletions(-) diff --git a/docs/source/user-guide/sql/timestamps.md b/docs/source/user-guide/sql/timestamps.md index 12017f60553a2..56236865cd680 100644 --- a/docs/source/user-guide/sql/timestamps.md +++ b/docs/source/user-guide/sql/timestamps.md @@ -19,49 +19,55 @@ # Timestamps and Time Zones -Almost every surprising result involving timestamps in DataFusion comes down to a -single question: when a timestamp gains or loses a time zone, does the _instant_ -stay the same or does the _wall clock_ stay the same? This page answers that -question once, and then applies the answer to casts, the session time zone, the -date/time functions, daylight saving time, and a handful of recipes. +One question controls almost all timestamp results in DataFusion. When +DataFusion adds a time zone to a timestamp, or removes one, what stays the same? +Is it the _instant_, or is it the _wall clock_? -All output on this page was produced with `datafusion-cli` on DataFusion 55.0.0 -with default settings unless a `SET` statement says otherwise. +This page gives the answer. Then it uses the answer for casts, the session time +zone, the date and time functions, daylight saving time, and some examples. + +This page uses two names: + +- An **aware timestamp** has a time zone. +- A **naive timestamp** has no time zone. + +`datafusion-cli` on DataFusion 55.0.0 made all the output on this page. The +settings are the default settings, unless a `SET` statement shows a different +setting. ## The data model -DataFusion timestamps are Arrow timestamps. Arrow has exactly two timestamp -shapes, and the difference between them is the whole story: +DataFusion timestamps are Arrow timestamps. Arrow has two timestamp types. The +difference between the two types controls all the behavior on this page. -| Arrow type | Physical value | Meaning | -| --------------------------- | ------------------------- | --------------------------------------------------------------------------------------------------------- | -| `Timestamp(unit, Some(tz))` | offset from the UTC epoch | An **instant**. `tz` is a display annotation: it says how to render the instant, not what the instant is. | -| `Timestamp(unit, None)` | a wall-clock reading | A **wall clock** with no instant attached. There is no fact about which point in time it names. | +| Arrow type | Physical value | Meaning | +| --------------------------- | ------------------------- | ------------------------------------------------------------------------------------------------------- | +| `Timestamp(unit, Some(tz))` | offset from the UTC epoch | An **instant**. `tz` is a display annotation: it says how to show the instant, not what the instant is. | +| `Timestamp(unit, None)` | a wall-clock reading | A **wall clock** with no instant attached. There is no fact about which point in time it names. | -Two consequences follow, and they explain most of the rest of this page. +Two results follow. These two results explain most of this page. -- A zone-aware timestamp's `tz` **is not data**. Two values with the same - integer and different `tz` annotations are the _same instant_; comparing, - sorting, grouping or joining them treats them as equal. -- A zone-naive timestamp has **no instant**, so any operation that needs one has - to invent a zone. Which zone gets invented is the subject of - [The session time zone](#the-session-time-zone) below, and it is not always - the same zone. +- The `tz` of an aware timestamp **is not data**. Two values with the same + integer and different `tz` annotations are the _same instant_. In a + comparison, a sort, a `GROUP BY` or a join, they are equal. +- A naive timestamp has **no instant**. An operation that needs an instant must + select a time zone. [The session time zone](#the-session-time-zone) shows + which time zone DataFusion selects. The time zone is not always the same. -### How SQL types map to Arrow types +### How SQL types agree with Arrow types | SQL type | Arrow type | | ---------------------------------------------------------- | ------------------------------------------------------------- | | `TIMESTAMP`, `TIMESTAMP WITHOUT TIME ZONE`, `::timestamp` | `Timestamp(Nanosecond, None)` | | `TIMESTAMP WITH TIME ZONE`, `TIMESTAMPTZ`, `::timestamptz` | `Timestamp(Nanosecond, Some(datafusion.execution.time_zone))` | -`TIMESTAMP(p)` with `p` of 0, 3, 6 or 9 selects second, millisecond, -microsecond or nanosecond precision respectively. +In `TIMESTAMP(p)`, a `p` of 0, 3, 6 or 9 selects second, millisecond, +microsecond or nanosecond precision. -The second row is the important one, and it is where DataFusion parts company -with PostgreSQL. `TIMESTAMP WITH TIME ZONE` resolves to whatever -`datafusion.execution.time_zone` is set to — and that setting **defaults to -unset**. With it unset, `TIMESTAMP WITH TIME ZONE` is a zone-_naive_ type: +The second row is important, because DataFusion and PostgreSQL do not agree +here. `TIMESTAMP WITH TIME ZONE` uses the value of +`datafusion.execution.time_zone`. The default value of that setting is **not +set**. If you do not set it, `TIMESTAMP WITH TIME ZONE` is a naive type: ```sql SELECT arrow_typeof('2024-01-01T12:00:00Z'::timestamptz) AS type, @@ -76,10 +82,10 @@ SELECT arrow_typeof('2024-01-01T12:00:00Z'::timestamptz) AS type, +---------------+---------------------+ ``` -The `Z` was accepted and then discarded. PostgreSQL has no equivalent state: its -`TimeZone` parameter is always set to something, so `timestamptz` is always a -zone-aware type there. Set the session time zone and the same query behaves the -way a PostgreSQL user expects: +DataFusion accepted the `Z` and then removed it. PostgreSQL cannot be in this +condition, because its `TimeZone` parameter always has a value. As a result, +`timestamptz` in PostgreSQL is always an aware type. Set the session time zone. +Then the same query gives an aware value, as in PostgreSQL: ```sql SET datafusion.execution.time_zone = 'America/Denver'; @@ -96,22 +102,38 @@ SELECT arrow_typeof('2024-01-01T12:00:00Z'::timestamptz) AS type, +---------------------------------+---------------------------+ ``` +DuckDB and PostgreSQL agree with each other here. Each of the two systems +always has a session time zone, and the default value is the local time zone of +the machine. As a result, `TIMESTAMPTZ` in those systems is always an aware +type: + +| System | Default session time zone | `'2024-01-01T12:00:00Z'::timestamptz` | +| ---------- | ------------------------- | ------------------------------------------- | +| DataFusion | not set | `Timestamp(ns)` — naive, the `Z` is removed | +| PostgreSQL | the machine's time zone | `timestamp with time zone` — aware | +| DuckDB | the machine's time zone | `TIMESTAMP WITH TIME ZONE` — aware | + +There is also a difference in the data model. In DuckDB, `TIMESTAMP WITH TIME ZONE` is one type, and a value of that type has no time zone of its own. The +session time zone controls the display of each value. In DataFusion, each value +keeps its own time zone in its Arrow type. DataFusion can hold two values in +two different time zones in the same query. DuckDB cannot do this. + :::{note} -If you are writing SQL that must behave predictably, set -`datafusion.execution.time_zone` explicitly. `'UTC'` is a good default: -it makes `timestamptz` a genuinely zone-aware type, makes `now()` zone-aware, -and keeps every conversion on this page a no-op shift. +Set `datafusion.execution.time_zone` if the time zone of your results is +important. `'UTC'` is a good value. It makes `timestamptz` an aware type, it +makes `now()` aware, and it gives each conversion on this page an offset of +zero. ::: ## The one rule -Every conversion between the two shapes follows from one rule, applied in three +One rule controls each conversion between the two types. The rule has three directions. -### Zone-naive to zone-aware: a **shift** +### From naive to aware: a **shift** -The naive wall clock is read as a local time _in the target zone_. The wall -clock is preserved; the instant changes. +DataFusion reads the naive wall clock as a local time _in the target time zone_. +The wall clock stays the same. The instant changes. ```sql SELECT @@ -131,13 +153,13 @@ SELECT +---------------------+---------------------------+-------------+-------------+ ``` -The wall clock is still `12:00:00`; the epoch moved by 25200 seconds, the -`-07:00` offset. This is the direction that trips people up, because the -underlying integer changed even though nothing about the printed value did. +The wall clock stays `12:00:00`. But the epoch moved 25200 seconds, which is +the `-07:00` offset. This direction causes many errors, because the integer +changes although the printed value does not change. -### Zone-aware to zone-aware: a **relabel** +### From aware to aware: a **relabel** -The instant is preserved; only the display annotation changes. +The instant stays the same. Only the display annotation changes. ```sql CREATE OR REPLACE VIEW utc AS @@ -159,11 +181,11 @@ FROM utc; +----------------------+---------------------------+------------+--------------+ ``` -### Zone-aware to zone-naive: the **UTC** wall clock +### From aware to naive: the **UTC** wall clock -The zone annotation is dropped and the integer is kept, which means the -resulting wall clock is the value's wall clock **in UTC** — regardless of what -the source annotation was and regardless of the session time zone. +DataFusion removes the `tz` annotation and keeps the integer. As a result, the +new wall clock is the wall clock of the value **in UTC**. The source annotation +and the session time zone do not change this result. ```sql SELECT arrow_cast(t, 'Timestamp(Second, None)') AS zoned_to_naive FROM utc; @@ -186,30 +208,30 @@ SELECT arrow_cast(arrow_cast(t, 'Timestamp(Second, Some("America/Denver"))'), +---------------------+ ``` -Both give `12:00:00`, the UTC wall clock, even though the second value displays -as `05:00:00-07:00`. PostgreSQL instead converts to the _session_ time zone -here; see [Differences from PostgreSQL](#differences-from-postgresql). +The two results are `12:00:00`, which is the UTC wall clock. This is correct +although the second value shows `05:00:00-07:00`. PostgreSQL is different: it +converts to the _session_ time zone. Refer to +[Differences from PostgreSQL](#differences-from-postgresql). -To get a local wall clock instead, use -[`to_local_time`](scalar_functions.md#to_local_time) — see -[Recipes](#recipes). +Use [`to_local_time`](scalar_functions.md#to_local_time) to get a local wall +clock. Refer to [Examples](#examples). ### Summary -| Conversion | What is preserved | What changes | -| ------------------ | ----------------- | -------------------------- | -| naive → zoned | wall clock | the instant (shifted) | -| zoned → zoned | the instant | the display annotation | -| zoned → naive | the instant | becomes the UTC wall clock | +| Conversion | What stays the same | What changes | +| ------------------ | ------------------- | ----------------------------- | +| naive → aware | the wall clock | the instant (it shifts) | +| aware → aware | the instant | the display annotation | +| aware → naive | the instant | it becomes the UTC wall clock | ### `AT TIME ZONE` -`AT TIME ZONE` applies the same rule, chosen by the input's shape: +`AT TIME ZONE` uses the same rule. The type of the input selects the direction: -- On a zone-**naive** input it is the naive → zoned shift: the wall clock - is read as local time in the named zone. -- On a zone-**aware** input it is the zoned → zoned relabel: the instant is - kept and the display annotation is replaced. +- For a **naive** input, `AT TIME ZONE` does the naive-to-aware shift. It reads + the wall clock as a local time in the time zone that you give. +- For an **aware** input, `AT TIME ZONE` does the aware-to-aware relabel. It + keeps the instant and replaces the display annotation. ```sql CREATE OR REPLACE VIEW utc AS @@ -239,21 +261,23 @@ FROM utc; +---------------------------------+---------------------------+---------------------+ ``` -The zone-naive case agrees with PostgreSQL: both name the instant -`2024-01-01T19:00:00Z`, DataFusion displaying it in Denver and PostgreSQL in the -session zone. +For a naive input, DataFusion and PostgreSQL agree. The two systems give the +instant `2024-01-01T19:00:00Z`. DataFusion shows it in Denver, and PostgreSQL +shows it in the session time zone. -The zone-aware case does not. PostgreSQL and DuckDB both return a zone-**naive** -`2024-01-01 05:00:00` there, so `(t AT TIME ZONE 'America/Denver')::timestamp` -gives `05:00:00` in those systems and `12:00:00` — the UTC wall clock — in -DataFusion. See ; this is -under active discussion and may change. +For an aware input, the two systems do not agree. PostgreSQL and DuckDB give a +**naive** `2024-01-01 05:00:00`. As a result, +`(t AT TIME ZONE 'America/Denver')::timestamp` gives `05:00:00` in those +systems. In DataFusion, it gives `12:00:00`, which is the UTC wall clock. Refer +to . This behavior can +change. -### Where the rule is not applied consistently +### Places where DataFusion does not use the rule -The rule above describes the cast kernel. Some parts of DataFusion take the -other reading — that a naive value is "really" UTC and gaining a zone is a -relabel — and that inconsistency is visible from SQL today. The clearest case: +The rule above tells you how the cast kernel operates. But some parts of +DataFusion use a different rule. In that different rule, a naive value is a UTC +value, and a new time zone is only a relabel. You can see this difference from +SQL today. This is the most clear example: ```sql CREATE OR REPLACE TABLE c AS SELECT TIMESTAMP '2024-01-01 12:00:00' AS ts; @@ -289,37 +313,45 @@ SELECT ts = arrow_cast(1704135600, 'Timestamp(Second, Some("America/Denver"))') +------------+ ``` -The cast in the first query shifts. The comparison in the third query is -rewritten by the `unwrap_cast_in_comparison` optimizer rule, which removes the -cast and keeps the literal's integer unchanged — a relabel. Note that -`datafusion.execution.time_zone` is not set here and plays no part: the two -queries disagree purely because one path shifts and the other relabels. Until - is fixed, prefer to make the -conversion explicit and to compare zone-aware values against zone-aware values. +The cast in the first query shifts the value. The `unwrap_cast_in_comparison` +optimizer rule changes the comparison in the third query. That rule removes the +cast and keeps the integer of the literal, which is a relabel. Note that +`datafusion.execution.time_zone` has no value here, and it has no effect in this +example. The two queries disagree only because one path shifts and the other +path relabels. + +Until there is a correction for +, obey these two rules: + +- Write each conversion in the SQL with `arrow_cast`. +- Compare aware values only with other aware values. ## The session time zone -`datafusion.execution.time_zone` (see [Configuration Settings](../configs.md)) -is the zone DataFusion uses when it has to invent one. It **defaults to unset**. +`datafusion.execution.time_zone` is the time zone that DataFusion uses when it +must select one. Refer to [Configuration Settings](../configs.md). The default +value is **not set**. -What it affects: +This setting changes: -- The Arrow type that `TIMESTAMP WITH TIME ZONE` / `::timestamptz` resolves to. -- The type and value returned by `now()`, `current_timestamp`, `current_date` - and `current_time`. -- The zone that `to_timestamp` and the `to_timestamp_*` family produce, and the - zone that a zone-less string argument to them is interpreted in. +- The Arrow type of `TIMESTAMP WITH TIME ZONE` and `::timestamptz`. +- The type and the value of `now()`, `current_timestamp`, `current_date` and + `current_time`. +- The time zone that `to_timestamp` and the `to_timestamp_*` functions give. +- The time zone that these functions use for a string argument that has no time + zone. -What it does **not** affect: +This setting does not change: -- The meaning of a `Timestamp(unit, Some(tz))` value that already exists. A - column read from Parquet or created by `AT TIME ZONE` keeps its own zone. -- `from_unixtime`, which always produces a zone-naive UTC wall clock - (). -- `date_part` / `EXTRACT` on a zone-naive value, which reports that value's - stored wall clock and never reinterprets it - (). -- Zone-aware → zone-naive casts, which always produce the UTC wall clock. +- The meaning of a `Timestamp(unit, Some(tz))` value that is in memory. A + column from a Parquet file, or a column from `AT TIME ZONE`, keeps its own + time zone. +- `from_unixtime`, which always gives the UTC wall clock as a naive value. Refer + to . +- `date_part` and `EXTRACT` on a naive value. These functions give the wall + clock that is in memory, and they never read it again in a different time + zone. Refer to . +- A cast from aware to naive, which always gives the UTC wall clock. ```sql SET datafusion.execution.time_zone = 'America/Denver'; @@ -347,17 +379,19 @@ SELECT to_timestamp('2024-01-01T12:00:00') AS to_timestamp_naive_string, +---------------------------+---------------------+ ``` -Note that `to_timestamp` read the zone-less string `'2024-01-01T12:00:00'` as -`12:00` _local_ Denver time, while `from_unixtime` returned a naive UTC wall -clock for the same epoch. +Note the difference. `to_timestamp` read the string `'2024-01-01T12:00:00'` as +`12:00` _local_ Denver time. But `from_unixtime` gave the UTC wall clock as a +naive value for the same epoch. -### The default-unset footgun +### The risk when the session time zone has no value -With `datafusion.execution.time_zone` unset, `::timestamptz` **erases** an -existing zone rather than converting to one, because the target type is -`Timestamp(_, None)` and the conversion is the zone-aware → zone-naive -rule: keep the integer, drop the annotation. Wrapping a correct expression in -`::timestamptz` therefore silently changes its meaning to UTC: +If `datafusion.execution.time_zone` has no value, `::timestamptz` **removes** +the time zone of an aware value. It does not convert to a time zone. The cause +is the target type, which is `Timestamp(_, None)`. DataFusion uses the +aware-to-naive rule: it keeps the integer, and it removes the annotation. + +If you put `::timestamptz` around a correct expression, you change the meaning +of that expression to UTC. There is no warning: ```sql CREATE OR REPLACE TABLE hits(t TIMESTAMP) AS VALUES @@ -392,53 +426,54 @@ FROM hits_utc GROUP BY 1 ORDER BY 1; +---------------------+---+ ``` -The second query groups by UTC day even though it names Brussels. See -. +The second query groups by the UTC day, although the SQL gives the name of +Brussels. Refer to . -### Combining values from different zones +### How to combine values from different time zones -When two timestamps of different types meet — in a comparison, a `UNION`, a -`CASE`, a `coalesce` — DataFusion picks a common type: +You can put two timestamps of different types together in a comparison, a +`UNION`, a `CASE` or a `coalesce`. DataFusion then selects a common type: | Left | Right | Common type | | ----------- | ------------- | ------------------------------------------ | | `Some(tz)` | the same `tz` | that `tz` | -| `Some(tz1)` | `Some(tz2)` | `Some("UTC")` (both are relabelled) | +| `Some(tz1)` | `Some(tz2)` | `Some("UTC")` (both are relabeled) | | `None` | `Some(tz)` | `Some(tz)` — the naive side is **shifted** | -Because the common type of two _different_ named zones is UTC and not the -session zone, a `UNION` of Denver and Brussels data displays as UTC. Nothing is -lost: all three cases preserve every instant. - -## Local time versus the instant - -Some functions work on the value's **local wall clock** — the reading you get -after applying the value's own `tz` annotation. Others work on the **UTC -instant** and ignore the annotation except for display. Which group a function -falls into is rarely stated anywhere, and it is the difference between a correct -and an incorrect query, so here it is explicitly. - -For a zone-aware value, "local" means the value's _own_ zone, not the session -zone. For a zone-naive value there is no zone to apply: the local-wall-clock -functions use the stored reading as-is, and the instant-based functions treat it -as UTC. - -| Function / operator | Operates on | Notes | -| ------------------------------------------------------------------- | ------------------- | ---------------------------------------------------------------------------------- | -| [`date_trunc`](scalar_functions.md#date_trunc), `datetrunc` | local wall clock | Truncates to local midnight, local hour, … | -| [`date_part`](scalar_functions.md#date_part), `datepart`, `EXTRACT` | local wall clock | | -| [`to_char`](scalar_functions.md#to_char), `date_format` | local wall clock | | -| [`to_local_time`](scalar_functions.md#to_local_time) | local wall clock | Returns `Timestamp(unit, None)` holding the local reading | -| `CAST(t AS DATE)`, `CAST(t AS TIME)` | local wall clock | | -| `timestamp + interval`, `timestamp - interval` | local wall clock | Calendar units are DST-aware; see below | -| [`generate_series`](scalar_functions.md#generate_series) / `range` | local wall clock | Steps by calendar units in the value's own zone; accepts nanosecond precision only | -| [`date_bin`](scalar_functions.md#date_bin) | **the UTC instant** | Bins are anchored at the UTC epoch, not at local midnight | -| [`to_unixtime`](scalar_functions.md#to_unixtime) | **the UTC instant** | | -| `AT TIME ZONE`, `CAST(t AS TIMESTAMPTZ)` | **the UTC instant** | For a zone-aware input; a zone-naive input is shifted, see above | -| Comparison, `ORDER BY`, `GROUP BY`, joins, `min`/`max` | **the UTC instant** | Annotations are ignored; equal instants are equal | - -The consequence worth memorising is that `date_trunc` and `date_bin` disagree on -the same value: +The common type of two _different_ time zones is UTC. It is not the session time +zone. As a result, a `UNION` of Denver data and Brussels data shows UTC. There +is no loss of data, because all three rows keep each instant. + +## Local time and the instant + +Some functions use the **local wall clock** of the value. The local wall clock +is the reading of the instant in the `tz` of the value. Other functions use the +**UTC instant**, and they use the annotation only for display. + +The group of a function is important. It makes the difference between a correct +query and an incorrect query. This page gives the group of each function. + +For an aware value, "local" is the _own_ time zone of the value. It is not the +session time zone. A naive value has no time zone. For a naive value, the local +wall clock is the reading that is in memory. The UTC instant is that same +reading as UTC. + +| Function / operator | Operates on | Notes | +| ------------------------------------------------------------------- | ------------------- | ------------------------------------------------------------------------------------ | +| [`date_trunc`](scalar_functions.md#date_trunc), `datetrunc` | local wall clock | Truncates to local midnight, local hour, … | +| [`date_part`](scalar_functions.md#date_part), `datepart`, `EXTRACT` | local wall clock | | +| [`to_char`](scalar_functions.md#to_char), `date_format` | local wall clock | | +| [`to_local_time`](scalar_functions.md#to_local_time) | local wall clock | Returns `Timestamp(unit, None)` that holds the local reading | +| `CAST(t AS DATE)`, `CAST(t AS TIME)` | local wall clock | | +| `timestamp + interval`, `timestamp - interval` | local wall clock | Calendar units obey DST; refer to the section below | +| [`generate_series`](scalar_functions.md#generate_series) / `range` | local wall clock | Steps by calendar units in the own time zone of the value; nanosecond precision only | +| [`date_bin`](scalar_functions.md#date_bin) | **the UTC instant** | The bin origin is the UTC epoch, not local midnight | +| [`to_unixtime`](scalar_functions.md#to_unixtime) | **the UTC instant** | | +| `AT TIME ZONE`, `CAST(t AS TIMESTAMPTZ)` | **the UTC instant** | For an aware input; DataFusion shifts a naive input, see above | +| Comparison, `ORDER BY`, `GROUP BY`, joins, `min`/`max` | **the UTC instant** | The annotations have no effect; equal instants are equal | + +Remember this result: `date_trunc` and `date_bin` do not agree on the same +value. ```sql CREATE OR REPLACE VIEW v AS @@ -460,12 +495,14 @@ FROM v; ``` `date_trunc` gave local midnight in Denver. `date_bin` gave the start of the UTC -day, displayed in Denver (`2023-12-31T17:00:00-07:00` is `2024-01-01T00:00:00Z`). -Each individually matches PostgreSQL; the pair is still surprising. Use -`date_trunc` when you want local calendar boundaries, and see -[Recipes](#recipes) for local-calendar binning with `date_bin`. +day, and it shows that instant in Denver. The value `2023-12-31T17:00:00-07:00` +is the same instant as `2024-01-01T00:00:00Z`. + +Each function agrees with PostgreSQL. But the difference between the two +functions is not easy to see. Use `date_trunc` for local calendar limits. For +local calendar bins with `date_bin`, refer to [Examples](#examples). -The other functions on a zone-aware value: +These are the other functions on an aware value: ```sql SELECT date_part('hour', t) AS date_part_hour, @@ -485,12 +522,15 @@ FROM v; ## Daylight saving time -### `INTERVAL '1 day'` is not `INTERVAL '24 hours'` +### `INTERVAL '1 day'` is different from `INTERVAL '24 hours'` -This is the single most useful thing to know on this page. On a **zone-aware** -timestamp, DataFusion adds calendar units (years, months, days) in the value's -own local calendar and adds sub-day units (hours, minutes, seconds) as elapsed -time. Across a DST transition the two differ: +This is the most important rule on this page. For an **aware** timestamp, +DataFusion adds calendar units in the local calendar of the value. Calendar +units are years, months and days. + +DataFusion adds units that are less than one day as elapsed time. These units +are hours, minutes and seconds. At a DST transition, the two types of unit give +different results: ```sql CREATE OR REPLACE VIEW d AS @@ -513,26 +553,30 @@ FROM d; +---------------------------+---------------------------+---------------------------+---------------+------------------+ ``` -America/Denver springs forward on 2024-03-10, so that local day is 23 hours -long. `INTERVAL '1 day'` lands on the same wall clock the next day (82800 -seconds later); `INTERVAL '24 hours'` lands 24 hours later, an hour further on -the clock. In the autumn the same pair goes the other way: adding -`INTERVAL '1 day'` to `2024-11-02 12:00:00-06:00` advances the instant by 90000 +The clocks in America/Denver move forward on 2024-03-10. As a result, that local +day has 23 hours. `INTERVAL '1 day'` gives the same wall clock on the next day, which is +82800 seconds later. `INTERVAL '24 hours'` gives an instant 24 hours later, +which is one hour later on the clock. + +In the autumn, the same two units give the opposite result. If you add +`INTERVAL '1 day'` to `2024-11-02 12:00:00-06:00`, the instant moves 90000 seconds. -This matches PostgreSQL exactly (verified against PostgreSQL 17). On a -**zone-naive** timestamp there is no DST to apply, so `INTERVAL '1 day'` and -`INTERVAL '24 hours'` always agree. +This behavior agrees with PostgreSQL 17. A **naive** timestamp has no DST. For a +naive timestamp, `INTERVAL '1 day'` and `INTERVAL '24 hours'` always give the +same result. -Use calendar units when you mean "the same time tomorrow" and sub-day units when -you mean "24 hours of elapsed time". They are not interchangeable. +Use calendar units for "the same time tomorrow". Use units of less than one day +for "24 hours of elapsed time". The two types of unit are not the same. -### Ambiguous and nonexistent local times +### Local times that are ambiguous or do not exist -Two local wall clocks per year are not well defined in a zone with DST: the hour -skipped when the clocks go forward does not exist, and the hour repeated when -they go back is ambiguous. DataFusion currently **errors** on both, in both the -literal and the column path: +In a time zone with DST, two local wall clocks each year do not identify one +instant. The hour that the clocks skip in the spring does not exist. The hour +that the clocks repeat in the autumn is ambiguous. + +DataFusion gives an **error** for the two hours. It gives an error for a literal +and also for a column: ```sql SET datafusion.execution.time_zone = 'America/Denver'; @@ -555,21 +599,26 @@ caused by Arrow error: Parser error: Error parsing timestamp from '2024-03-10T02:30:00': error computing timezone offset ``` -The same happens for the ambiguous fall-back hour, `2024-11-03 01:30:00` in -America/Denver. PostgreSQL resolves both instead of erroring (the gap moves -forward, the ambiguous hour is read as standard time). +The ambiguous hour in the autumn gives the same errors. In America/Denver, that +hour is `2024-11-03 01:30:00`. PostgreSQL does not give an error. It gives a +result for the two hours: it moves a time in the gap forward, and it reads the +ambiguous hour as standard time. + +A time zone with a fixed offset, such as `'+08:00'`, has no transitions. It +cannot give these errors. Refer to + and to the correction in +. -Fixed-offset zones such as `'+08:00'` never have transitions and are never -affected. See and the -upstream fix . +## Examples -## Recipes +### How to group UTC data by the local calendar day -### Aggregate UTC data by local calendar day in a named zone +`date_bin` uses the UTC instant. If you give aware data to `date_bin`, you get +UTC days. Do these three steps: -`date_bin` bins on the UTC instant, so binning zone-aware data directly gives -UTC days. Convert to the target zone, then flatten to a local wall clock with -`to_local_time`, and bin that: +1. Convert the data to the target time zone. +2. Make the result a local wall clock with `to_local_time`. +3. Give that local wall clock to `date_bin`. ```sql CREATE OR REPLACE TABLE hits(t TIMESTAMP) AS VALUES @@ -600,10 +649,11 @@ FROM hits_utc; +---------------------------+---------------------+ ``` -`local_day` is a zone-naive value: it is a local calendar day label, not an -instant, which is exactly what you want as a `GROUP BY` key. For whole calendar -units, `date_trunc('day', t AT TIME ZONE 'Europe/Brussels')` gives the same -grouping while keeping the result zone-aware: +`local_day` is a naive value. It is a label for a local calendar day, and it is +not an instant. This is the correct type for a `GROUP BY` key. + +For a whole calendar unit, `date_trunc('day', t AT TIME ZONE 'Europe/Brussels')` +gives the same groups. It also keeps the result aware: ```sql SELECT date_trunc('day', t AT TIME ZONE 'Europe/Brussels') AS day, count(*) AS n @@ -619,46 +669,48 @@ FROM hits_utc GROUP BY 1 ORDER BY 1; +---------------------------+---+ ``` -Use the `to_local_time` form when the bin width is not a whole calendar unit -(`INTERVAL '15 minutes'`, `INTERVAL '4 hours'`), which `date_trunc` cannot -express. +Use the `to_local_time` form if the width of the bin is not a whole calendar +unit. `date_trunc` cannot give `INTERVAL '15 minutes'` or `INTERVAL '4 hours'`. :::{warning} -Passing `date_bin` an origin in the target zone — for example -`date_bin(INTERVAL '1 day', t, TIMESTAMP '2024-04-30 22:00:00' AT TIME ZONE 'UTC')` -— appears to work but is only correct while the zone's offset does not change. -It silently drifts by an hour across a DST transition. Prefer `to_local_time`. +Do not give `date_bin` an origin in the target time zone. For example, do not +use +`date_bin(INTERVAL '1 day', t, TIMESTAMP '2024-04-30 22:00:00' AT TIME ZONE 'UTC')`. +The result is correct only while the offset of the time zone stays the same. At +a DST transition, the result moves one hour, and there is no warning. Use +`to_local_time` instead. ::: -### Get a zone's local wall clock +### How to find the local wall clock of a time zone ```sql SELECT to_local_time(t AT TIME ZONE 'Europe/Brussels') AS brussels_wall_clock FROM hits_utc LIMIT 1; ``` -`AT TIME ZONE` relabels the instant for display in Brussels; `to_local_time` -then converts that display into an actual zone-naive value. Do **not** use -`::timestamp` for this — that returns the UTC wall clock, not the local one. +`AT TIME ZONE` relabels the instant for display in Brussels. Then +`to_local_time` makes a naive value from that display. Do **not** use +`::timestamp` for this task. `::timestamp` gives the UTC wall clock, not the +local wall clock. -### Round-trip safely +### How to convert safely -- Store instants as `Timestamp(unit, Some("UTC"))` and convert to a display zone - only at the edge of the query. -- Set `datafusion.execution.time_zone = 'UTC'` so that `timestamptz`, `now()` - and `to_timestamp` are zone-aware and every conversion in this page is a - zero-offset shift. -- Use `arrow_cast` with an explicit Arrow type when you want the exact - conversion; `::timestamptz` depends on session configuration. -- Never round-trip a zone-aware value through `Timestamp(_, None)` unless you - intend to reduce it to a UTC wall clock. +- Keep instants as `Timestamp(unit, Some("UTC"))`. Convert to a display time + zone only at the end of the query. +- Set `datafusion.execution.time_zone = 'UTC'`. Then `timestamptz`, `now()` and + `to_timestamp` are aware, and each conversion on this page has an offset of + zero. +- Use `arrow_cast` with an Arrow type if you want an exact conversion. The + result of `::timestamptz` changes with the session configuration. +- Do not convert an aware value through `Timestamp(_, None)`, unless you want to + make it a UTC wall clock. ## Differences from PostgreSQL -DataFusion aims to follow PostgreSQL, and does for most of the above. These are -the known divergences today. +DataFusion follows PostgreSQL for most of the behavior above. These are the +known differences today. -| Behaviour | DataFusion | PostgreSQL | Issue | +| Behavior | DataFusion | PostgreSQL | Issue | | -------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------- | --------------------------------------------------- | | `TIMESTAMP WITH TIME ZONE` with no session zone set | Zone-**naive** `Timestamp(_, None)`; the offset in a literal is discarded | Always zone-aware; `TimeZone` is always set | — | | `AT TIME ZONE` applied to a zone-**aware** value | Returns a zone-**aware** value in the named zone | Returns a zone-**naive** value | | @@ -672,12 +724,13 @@ the known divergences today. | `to_timestamp_*` on an already zone-aware input | Rewrites the zone to the session zone, dropping it entirely when unset | n/a | | :::{note} -The `AT TIME ZONE` row describes DataFusion's behaviour as of this writing. There -is active discussion in -about aligning the zone-**aware**-input case with PostgreSQL and DuckDB, which -would change the second and third rows above; DuckDB also returns a zone-naive -value there. The zone-**naive** input case already agrees with PostgreSQL: both -produce the same instant. +The `AT TIME ZONE` row shows the behavior of DataFusion today. The DataFusion +community discusses a change in +. If the community makes that +change, the **aware** input will agree with PostgreSQL and DuckDB. The second +row and the third row above will also change. DuckDB also gives a naive value +there. The **naive** input agrees with PostgreSQL now, because the two systems +give the same instant. ::: ## See also From e0522fa1e3d3b56b40b2c4102bed9fdb1f7bf52d Mon Sep 17 00:00:00 2001 From: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com> Date: Thu, 10 Sep 2026 12:41:34 -0500 Subject: [PATCH 3/4] docs: use one timestamp vocabulary across the whole timestamps page The Simplified Technical English pass defined "aware timestamp" and "naive timestamp" as the two terms for the page, but the PostgreSQL divergence table kept the older "zone-aware"/"zone-naive" wording, so the page carried two vocabularies for one concept. STE requires one term per concept. Rewords those table cells and one SQL comment. No factual claim changes, and every query and result block is byte-identical (verified block by block). Co-Authored-By: Claude Opus 5 --- docs/source/user-guide/sql/timestamps.md | 36 ++++++++++++------------ 1 file changed, 18 insertions(+), 18 deletions(-) diff --git a/docs/source/user-guide/sql/timestamps.md b/docs/source/user-guide/sql/timestamps.md index 56236865cd680..8df09f453581a 100644 --- a/docs/source/user-guide/sql/timestamps.md +++ b/docs/source/user-guide/sql/timestamps.md @@ -107,11 +107,11 @@ always has a session time zone, and the default value is the local time zone of the machine. As a result, `TIMESTAMPTZ` in those systems is always an aware type: -| System | Default session time zone | `'2024-01-01T12:00:00Z'::timestamptz` | -| ---------- | ------------------------- | ------------------------------------------- | -| DataFusion | not set | `Timestamp(ns)` — naive, the `Z` is removed | -| PostgreSQL | the machine's time zone | `timestamp with time zone` — aware | -| DuckDB | the machine's time zone | `TIMESTAMP WITH TIME ZONE` — aware | +| System | Default time zone of the session | `'2024-01-01T12:00:00Z'::timestamptz` | +| ---------- | -------------------------------- | ------------------------------------------- | +| DataFusion | not set | `Timestamp(ns)` — naive, the `Z` is removed | +| PostgreSQL | the machine's time zone | `timestamp with time zone` — aware | +| DuckDB | the machine's time zone | `TIMESTAMP WITH TIME ZONE` — aware | There is also a difference in the data model. In DuckDB, `TIMESTAMP WITH TIME ZONE` is one type, and a value of that type has no time zone of its own. The session time zone controls the display of each value. In DataFusion, each value @@ -401,7 +401,7 @@ CREATE OR REPLACE TABLE hits(t TIMESTAMP) AS VALUES (TIMESTAMP '2024-05-01T20:30:00'); CREATE OR REPLACE VIEW hits_utc AS SELECT t AT TIME ZONE 'UTC' AS t FROM hits; --- correct: group on the zone-aware value +-- correct: group on the aware value SELECT date_trunc('day', t AT TIME ZONE 'Europe/Brussels') AS day, count(*) AS n FROM hits_utc GROUP BY 1 ORDER BY 1; @@ -710,18 +710,18 @@ local wall clock. DataFusion follows PostgreSQL for most of the behavior above. These are the known differences today. -| Behavior | DataFusion | PostgreSQL | Issue | -| -------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------- | --------------------------------------------------- | -| `TIMESTAMP WITH TIME ZONE` with no session zone set | Zone-**naive** `Timestamp(_, None)`; the offset in a literal is discarded | Always zone-aware; `TimeZone` is always set | — | -| `AT TIME ZONE` applied to a zone-**aware** value | Returns a zone-**aware** value in the named zone | Returns a zone-**naive** value | | -| `tstz::timestamp` | The **UTC** wall clock | The **session zone** wall clock | | -| Zone-naive value compared with a zone-aware one | The naive side is shifted into the **other operand's** zone, not the session zone | The naive side is read in the **session** zone | | -| Zone-naive **column** compared with a zone-aware literal | The optimizer drops the shift, giving the opposite answer to the same comparison written with literals | Consistent with the literal form | | -| `tstz - timestamp` | Naive side read as UTC, so the session offset is lost | Naive side read in the session zone | | -| Ambiguous / nonexistent local time | Error | Resolved (gap moves forward, ambiguity reads as standard time) | | -| `from_unixtime` | Zone-naive UTC wall clock; ignores the session zone | `to_timestamp(double)` returns `timestamptz` | | -| `date_part` / `EXTRACT` on a zone-naive value | Uses the stored wall clock; ignores the session zone | The same — but DataFusion's config docs once promised session-zone awareness | | -| `to_timestamp_*` on an already zone-aware input | Rewrites the zone to the session zone, dropping it entirely when unset | n/a | | +| Behavior | DataFusion | PostgreSQL | Issue | +| ------------------------------------------------------ | ---------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------- | --------------------------------------------------- | +| `TIMESTAMP WITH TIME ZONE` with no session zone set | **Naive** `Timestamp(_, None)`; DataFusion removes the offset in a literal | Always **aware**; `TimeZone` always has a value | — | +| `AT TIME ZONE` used on an **aware** value | Gives an **aware** value in the given time zone | Gives a **naive** value | | +| `tstz::timestamp` | The **UTC** wall clock | The **session zone** wall clock | | +| **Naive** value compared with an **aware** value | DataFusion shifts the naive side into the zone of the **other operand**, not the session zone | PostgreSQL reads the naive side in the **session** zone | | +| **Naive** column compared with an **aware** literal | The optimizer removes the shift. The answer is the opposite of the same comparison with two literals | The same answer as the literal form | | +| `tstz - timestamp` | Naive side read as UTC, so the session offset is lost | Naive side read in the session zone | | +| Ambiguous / nonexistent local time | Error | Resolved (gap moves forward, ambiguity reads as standard time) | | +| `from_unixtime` | Gives the UTC wall clock as a **naive** value. The session zone has no effect | `to_timestamp(double)` returns `timestamptz` | | +| `date_part` / `EXTRACT` on a **naive** value | Uses the wall clock in memory. The session zone has no effect | The same — but DataFusion's config docs once promised session-zone awareness | | +| `to_timestamp_*` on an input that is already **aware** | Replaces the zone with the session zone. If the session zone has no value, the zone is removed | n/a | | :::{note} The `AT TIME ZONE` row shows the behavior of DataFusion today. The DataFusion From e40156d75ce20063e77c6ca68e41bd3c4eccca7b Mon Sep 17 00:00:00 2001 From: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com> Date: Thu, 10 Sep 2026 13:07:09 -0500 Subject: [PATCH 4/4] docs: warn that the to_local_time round trip stops erroring across DST `to_local_time(t) AT TIME ZONE 'zone'` currently raises "Cannot cast timezone to different timezone" across an ambiguous hour, so the round trip cannot silently return a different instant. apache/arrow-rs#11038 removes that error, and once DataFusion picks it up the round trip moves by an hour with no error and no warning. PostgreSQL behaves the same way, so this is the intended trade rather than a regression -- but today the error is what stops users writing that round trip, so the docs need to say so before the behaviour changes. Co-Authored-By: Claude Opus 5 --- docs/source/user-guide/sql/timestamps.md | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/docs/source/user-guide/sql/timestamps.md b/docs/source/user-guide/sql/timestamps.md index 8df09f453581a..3cf92454c872d 100644 --- a/docs/source/user-guide/sql/timestamps.md +++ b/docs/source/user-guide/sql/timestamps.md @@ -609,6 +609,25 @@ cannot give these errors. Refer to and to the correction in . +:::{warning} +Do not use `to_local_time(t) AT TIME ZONE 'zone'` as a round trip across an +ambiguous hour. The result is correct today, because the second step gives an +error. But the correction in +removes that error. After DataFusion takes that correction, the round trip +gives an instant one hour from the first instant, and there is no error and no +warning: + +| Start value | After `to_local_time` | After the round trip | +| --------------------------- | --------------------- | --------------------------- | +| `2021-10-31T02:00:00+02:00` | `2021-10-31T02:00:00` | `2021-10-31T02:00:00+01:00` | +| `2021-10-31T02:30:00+02:00` | `2021-10-31T02:30:00` | `2021-10-31T02:30:00+01:00` | +| `2021-10-31T03:00:00+01:00` | `2021-10-31T03:00:00` | `2021-10-31T03:00:00+01:00` | + +The first two rows move. The third row does not move. PostgreSQL gives the same +results, because a local wall clock in the ambiguous hour cannot identify one +instant. Keep the aware value if you must have the instant. +::: + ## Examples ### How to group UTC data by the local calendar day