Skip to content

fix(lumina): reject non-finite vector elements when building an index - #898

Merged
JingsongLi merged 1 commit into
apache:mainfrom
jackylee-ch:fix/lumina-reject-non-finite-vectors
Sep 22, 2026
Merged

JingsongLi merged 1 commit into
apache:mainfrom
jackylee-ch:fix/lumina-reject-non-finite-vectors

Conversation

@jackylee-ch

Copy link
Copy Markdown
Contributor

extract_vectors_from_batches validated a null _ROW_ID, row-id contiguity, a null vector row, the dimension and a null element — but not finiteness, so a NaN or ±inf embedding was indexed silently. The everyday cause is a cosine normalisation dividing a zero vector by its zero norm.

Java rejects these per element in LuminaVectorGlobalIndexWriter#checkFinite, covered by testNanInVectorRejected and testInfinityInVectorRejected. The other Rust vector backend rejects them too, in paimon_vindex_core's validate_finite_values on both the write and the query side — so switching index.type between the two backends turned a loud build failure into a silent one.

The message carries the row id and the position inside the vector, as Java's does: a rejected build is only actionable if the offending row can be found.

extract_vectors_from_batches validated a null _ROW_ID, row-id contiguity,
a null vector row, the dimension and a null element, but not finiteness,
so a NaN or +-inf embedding was indexed silently.

Java rejects these per element in LuminaVectorGlobalIndexWriter#checkFinite,
called for every input shape and covered by testNanInVectorRejected and
testInfinityInVectorRejected. The other Rust vector backend rejects them
too, in paimon_vindex_core's validate_finite_values on both the write and
the query side -- so switching index.type between the two backends turned a
loud build failure into a silent one.

The message carries the row id and the position inside the vector, as Java's
does, since a rejected build is only actionable if the offending row can be
found.

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed head 8f21ee4. Requirement fit: supported; implementation: no blocking findings. The Lumina build path reads Arrow List and FixedSizeList through this shared element loop, and previously forwarded NaN/infinity to native index construction; rejecting the specific row ID and vector offset before commit makes that invalid input diagnosable and aligns with the existing vindex behavior. I ran both new targeted tests in an isolated build: test_extract_vectors_rejects_non_finite_element (NaN and ±infinity) and test_extract_vectors_fixed_size_list_rejects_non_finite_element; both passed. The check covers each non-null element without altering null/dimension/row-ID validation.

@JingsongLi
JingsongLi merged commit eb6a4f3 into apache:main Sep 22, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants