Skip to content

fix: canonicalize floating-point NaN values - #255

Merged
lxy-9602 merged 6 commits into
apache:mainfrom
zjw1111:codex/fix-floating-point-nan
Aug 28, 2026
Merged

fix: canonicalize floating-point NaN values#255
lxy-9602 merged 6 commits into
apache:mainfrom
zjw1111:codex/fix-floating-point-nan

Conversation

@zjw1111

@zjw1111 zjw1111 commented Aug 27, 2026

Copy link
Copy Markdown
Collaborator

Purpose

Linked issue: close #256

Canonicalize floating-point NaN payloads anywhere Java-compatible equality, hashing, or binary output requires a single representation.

This change centralizes float/double canonical bit conversion and applies it to:

  • Literal hashing, Bloom filter hashing, and Hive bucket hashing.
  • BTree key serialization.
  • Variant float and double serialization.
  • Global index result and indexed split score serialization.

Tests

  • pre-commit run --files <all changed files>
  • git diff --check
  • Compiled all newly changed production and test translation units with the repository Debug compile commands and -Werror.
  • MathTest.FloatingPointNaNCanonicalization
  • FastHashTest.TestNaNCompatibleWithJava
  • LiteralTest.*
  • HiveBucketFunctionTest.TestFloatNaNCanonicalizationCompatibleWithJava
  • HiveBucketFunctionTest.TestDoubleNaNCanonicalizationCompatibleWithJava
  • GenericVariantTest.CanonicalizesFloatingPointNaN
  • KeySerializerTest.CanonicalizesFloatingPointNaN
  • GlobalIndexResultTest.TestSerializeCanonicalizesNaNScore
  • IndexedSplitTest.TestSerializeCanonicalizesNaNScore

API and Format

No public API signature or protocol changes. Non-canonical NaN payloads are now written using the canonical Java-compatible NaN representation; existing canonical values and all non-NaN values are unchanged.

Documentation

No user-facing documentation changes.

Generative AI tooling

Generated-by: OpenAI Codex (GPT-5).

@zjw1111
zjw1111 marked this pull request as draft August 27, 2026 11:33
@zjw1111 zjw1111 changed the title fix(file-index): canonicalize floating-point NaN values fix: canonicalize floating-point NaN values Aug 27, 2026
@zjw1111
zjw1111 marked this pull request as ready for review August 28, 2026 02:48
Comment thread src/paimon/core/global_index/indexed_split_test.cpp Outdated
Comment thread src/paimon/core/global_index/indexed_split_test.cpp Outdated

@lxy-9602 lxy-9602 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@lxy-9602
lxy-9602 merged commit feb511e into apache:main Aug 28, 2026
28 of 29 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Canonicalize floating-point NaN values for Java compatibility

2 participants