Skip to content

[Python] to_pandas drops the time zone of a dictionary-encoded timestamp column #55490

Open

Description

@jmao-denver

Describe the bug, including details regarding any error messages, version, and platform.

to_pandas() drops the time zone of a dictionary-encoded timestamp column. The column becomes a pandas category whose categories are tz-naive timestamps. Comparing it with a tz-aware timestamp then returns False for every row, with no error:

import pandas as pd
import pyarrow as pa

ts = pa.array([0, 0, 10**9], type=pa.timestamp("ns", tz="UTC"))
plain = pa.table({"t": ts}).to_pandas()["t"]
encoded = pa.table({"t": ts.dictionary_encode()}).to_pandas()["t"]

print(plain.dtype)                       # datetime64[ns, UTC]
print(encoded.cat.categories.dtype)      # datetime64[ns], time zone dropped

t0 = pd.Timestamp(0, tz="UTC")
print((plain == t0).tolist())            # [True, True, False]
print((encoded == t0).tolist())          # [False, False, False]

The categories should keep the time zone, as datetime64[ns, UTC]. encoded > t0 and encoded.dt.tz_convert(...) also raise TypeError, while both work on plain.

Version: pyarrow 26.0.1, pandas 3.1.1, Python 3.15, macOS.

Component(s)

Python

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions