[GH-3609]: Add new sort order for int96 timestamps#3610
Open
divjotarora wants to merge 5 commits into
Open
Conversation
rdblue
reviewed
Jun 11, 2026
rdblue
reviewed
Jun 11, 2026
rdblue
reviewed
Jun 11, 2026
rdblue
reviewed
Jun 11, 2026
rdblue
reviewed
Jun 11, 2026
rdblue
reviewed
Jun 11, 2026
rdblue
reviewed
Jun 11, 2026
rdblue
reviewed
Jun 11, 2026
Member
|
Please resolve the conflicts. |
Contributor
Author
@wgtmac Sorry for the delay, I was on holiday last week. Conflicts are resolved now |
rdblue
reviewed
Jul 10, 2026
| * | ||
| * Two-level comparison, matching the INT96 timestamp sort order: | ||
| * 1. Compare the last 4 bytes (Julian day) as a signed little-endian int32. | ||
| * 2. If equal, compare the first 8 bytes (nanos) as a signed little-endian int64. |
Contributor
There was a problem hiding this comment.
The correctness of this depends on the two fields so I think we have to be more careful since the caller may pass in something unexpected.
I think that we need to validate that the nanosecond value is positive and less than the number of nanoseconds per day. If either of these is violated, then this sort order is no longer correct.
Contributor
Author
There was a problem hiding this comment.
Added validation for both cases
rdblue
reviewed
Jul 10, 2026
rdblue
reviewed
Jul 10, 2026
rdblue
reviewed
Jul 10, 2026
rdblue
reviewed
Jul 10, 2026
rdblue
reviewed
Jul 10, 2026
rdblue
reviewed
Jul 10, 2026
rdblue
reviewed
Jul 10, 2026
rdblue
reviewed
Jul 10, 2026
rdblue
reviewed
Jul 10, 2026
rdblue
reviewed
Jul 10, 2026
rdblue
reviewed
Jul 10, 2026
| * min/max statistics in place. This simulates a legacy writer that emitted INT96 stats under | ||
| * TYPE_ORDER before INT96_TIMESTAMP_ORDER existed; such stats must be ignored by the reader. | ||
| */ | ||
| private static void downgradeInt96ToTypeOrder(FileMetaData footer) { |
Contributor
There was a problem hiding this comment.
Can you add a couple more cases? I'd like to see UNDEFINED, TYPE_DEFINED_ORDER, and unset cases.
Contributor
Author
There was a problem hiding this comment.
Removed this helper and added both testReaderIgnoresInt96StatsWithTypeDefinedOrder and testReaderIgnoresInt96StatsWhenColumnOrdersAbsent
Contributor
Author
There was a problem hiding this comment.
I wasn't able to test UNDEFINED as I didn't find a good way to serialize an undefined value for the union
rdblue
reviewed
Jul 10, 2026
rdblue
reviewed
Jul 10, 2026
rdblue
reviewed
Jul 10, 2026
rdblue
reviewed
Jul 10, 2026
Contributor
|
I think there are a few things to be fixed, but overall I trust that this works. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
See parquet-format proposal for rationale: apache/parquet-format#584
What changes are included in this PR?
Add new
INT96_TIMESTAMP_ORDERsort order and related parsing.Are these changes tested?
Yes, new unit tests.
Are there any user-facing changes?
Two new flags and a Thrift change.
Closes #3609