fix(lumina): store the canonical native metric name in index metadata - #895
JingsongLi merged 1 commit into
Conversation
lumina.distance.metric accepts the enum spellings L2/COSINE/INNER_PRODUCT as well as the native ones, but the raw string was copied into the native option map that becomes the committed index metadata. The read side is exact-match only, so LuminaIndexMeta::metric() then failed with "Unknown lumina metric name: L2" on every query against such an index. Persist metric.lumina_name() instead, mirroring Java LuminaVectorIndexOptions (apache/paimon#8676). strip_lumina_options needs the same treatment. Java crosses its native boundary with a typed MetricType, so the string is inert there, but Rust hands the whole option map to the native library and ensure_loaded overlays the stripped table options on top of the index metadata -- the configured spelling would otherwise come straight back. An unrecognized value is left untouched so its error surface does not move, and the readers stay exact-match: from_lumina_name is the wire decoder and widening it would accept metadata Java rejects.
JingsongLi
left a comment
JingsongLi
left a comment
There was a problem hiding this comment.
Reviewed head b961ff6. Requirement fit: supported; implementation: no blocking findings. This is caller-visible: LuminaIndexBuildBuilder::execute commits to_lumina_options() as index metadata, while LuminaIndexMeta::metric() accepts only native spellings; both Lumina search paths and the DE/PK vector planners consume that parser. A configured L2 could therefore build successfully but make subsequent searches fail. Canonicalizing before metadata serialization fixes that without weakening the wire decoder, and canonicalizing stripped options avoids reintroducing the alias at the native reader boundary. In an isolated build, cargo test -p paimon --lib lumina::tests:: passed all 17 tests, including the enum/native spelling and unknown-value cases. Existing indexes whose metadata already contains the alias still need rebuilding, as the PR notes.
2 participants
lumina.distance.metricaccepts the enum spellings (L2) as well as the native ones, but the raw string was copied into the native option map thatLuminaIndexBuildBuildercommits as index metadata.LuminaIndexMeta::metric()decodes with exact-matchfrom_lumina_name, so every query against such an index failed withUnknown lumina metric name: L2. Fixed by persistingmetric.lumina_name(), as Java does (apache/paimon#8676).For reviewers: the
strip_lumina_optionscall is not redundant.LuminaVectorReader::ensure_loadedoverlays the stripped table options on top of the metadata before creating the searcher, so the configured spelling would otherwise be what crosses the FFI, not the one the index was built with.Readers stay exact-match and an unrecognized value passes through untouched, both pinned by tests. An index already committed with an enum spelling needs a rebuild.