fix(lumina): reject non-finite vector elements when building an index - #898
Merged
JingsongLi merged 1 commit intoSep 22, 2026
Merged
Conversation
extract_vectors_from_batches validated a null _ROW_ID, row-id contiguity, a null vector row, the dimension and a null element, but not finiteness, so a NaN or +-inf embedding was indexed silently. Java rejects these per element in LuminaVectorGlobalIndexWriter#checkFinite, called for every input shape and covered by testNanInVectorRejected and testInfinityInVectorRejected. The other Rust vector backend rejects them too, in paimon_vindex_core's validate_finite_values on both the write and the query side -- so switching index.type between the two backends turned a loud build failure into a silent one. The message carries the row id and the position inside the vector, as Java's does, since a rejected build is only actionable if the offending row can be found.
JingsongLi
reviewed
Sep 22, 2026
JingsongLi
left a comment
Contributor
There was a problem hiding this comment.
Reviewed head 8f21ee4. Requirement fit: supported; implementation: no blocking findings. The Lumina build path reads Arrow List and FixedSizeList through this shared element loop, and previously forwarded NaN/infinity to native index construction; rejecting the specific row ID and vector offset before commit makes that invalid input diagnosable and aligns with the existing vindex behavior. I ran both new targeted tests in an isolated build: test_extract_vectors_rejects_non_finite_element (NaN and ±infinity) and test_extract_vectors_fixed_size_list_rejects_non_finite_element; both passed. The check covers each non-null element without altering null/dimension/row-ID validation.
JingsongLi
approved these changes
Sep 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
extract_vectors_from_batchesvalidated a null_ROW_ID, row-id contiguity, a null vector row, the dimension and a null element — but not finiteness, so aNaNor±infembedding was indexed silently. The everyday cause is a cosine normalisation dividing a zero vector by its zero norm.Java rejects these per element in
LuminaVectorGlobalIndexWriter#checkFinite, covered bytestNanInVectorRejectedandtestInfinityInVectorRejected. The other Rust vector backend rejects them too, inpaimon_vindex_core'svalidate_finite_valueson both the write and the query side — so switchingindex.typebetween the two backends turned a loud build failure into a silent one.The message carries the row id and the position inside the vector, as Java's does: a rejected build is only actionable if the offending row can be found.