Skip to content

fix: convert nano timestamps to nanoseconds in partition representation - #3695

Merged
Fokko merged 1 commit into
apache:mainfrom
mittalpk:fix/partition-nano-timestamp-representation
Aug 31, 2026
Merged

fix: convert nano timestamps to nanoseconds in partition representation#3695
Fokko merged 1 commit into
apache:mainfrom
mittalpk:fix/partition-nano-timestamp-representation

Conversation

@mittalpk

Copy link
Copy Markdown
Contributor

Summary

_to_partition_representation() in pyiceberg/partitioning.py has no registered handler for TimestampNanoType/TimestamptzNanoType. A datetime partition value for a nano-precision timestamp column passes through unconverted instead of being converted to nanoseconds since epoch, the same way TimestampType/TimestamptzType already convert to microseconds.

That unconverted datetime object would reach DataFile.partition, where TimestampNanoWriter calls write_int — an int is expected, not a datetime.

Fixes #3652

Changes

  • Register a handler for TimestampNanoType/TimestamptzNanoType in _to_partition_representation, using the existing datetime_to_nanos() from pyiceberg/utils/datetime.py (mirrors the existing micros handler exactly).
  • Added test_to_partition_representation_timestamps (8 parametrized cases covering None, int passthrough, and datetime conversion for both the existing micros types and the new nano types) and test_to_partition_representation_unrecognized_type_raises (confirms unrecognized input types raise ValueError for both micros and nanos types — previously nano types silently fell through to the generic PrimitiveType handler, which returns the value unchanged rather than raising).

Test plan

  • New tests verified to fail against pre-fix code (git stash the fix in partitioning.py, rerun — 3 of 9 new tests fail as expected) and pass post-fix.
  • uv run pytest tests/test_transforms.py: 274 passed (up from 265 pre-fix, the +9 are the new tests); the 17 pre-existing failures (missing pyiceberg-core extension in this environment) are unchanged before/after — confirmed identical failure set on unmodified main.
  • uv run pytest tests/table/test_partitioning.py: 19 passed / 11 pre-existing failures (same extension-availability cause), unchanged before/after.
  • uv run ruff check / ruff format --check: clean on both changed files.
  • uv run mypy pyiceberg/partitioning.py: no errors attributable to this file (pre-existing unrelated errors in other files are due to optional dependency stubs not installed in this environment).

_to_partition_representation() had no handler for TimestampNanoType/
TimestamptzNanoType, so a datetime partition value passed through
unconverted instead of being converted to nanoseconds since epoch (the
same way TimestampType/TimestamptzType already convert to micros). The
resulting datetime object would reach DataFile.partition where an int
is expected, since TimestampNanoWriter calls write_int.

Fixes apache#3652
@github-actions

Copy link
Copy Markdown

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that's incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

@github-actions github-actions Bot added the stale label Aug 24, 2026
@mittalpk

Copy link
Copy Markdown
Contributor Author

Bumping to avoid the stale-bot auto-close — still awaiting review, happy to address any feedback.

@github-actions github-actions Bot removed the stale label Aug 25, 2026

@Fokko Fokko left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mittalpk Not much feedback from my end, this looks great! Thanks for picking this up 👍

@Fokko
Fokko merged commit d0b5894 into apache:main Aug 31, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Partition values for timestamp_ns columns are not converted to nanoseconds

2 participants