feat: add cosine and inner-product range search for IVF indexes

[JunRuiLee]

Summary

• Support cosine and inner-product range search for IVF-Flat, IVF-SQ, IVF-PQ and IVF-RQ through all four typed/unified single, batch and Roaring-filtered entry points.
• Define shared public score conversion, exact f64 predicate-to-f32 band conversion, IP side/operator reversal, signed-zero/extreme endpoint handling, and range capability queries.
• Preserve the membership contract: Flat uses exact stored-vector distances; SQ/PQ/RQ use their declared estimates. Full probing does not remove quantization error.
• Extend feat: add IVF-PQ L2 distance range search #107's native PQ scanner rather than adding a separate scanner. Preserve PQ4/PQ8, residual coding, OPQ, lazy bounded LUT caching, scratch fallback, parallel query scans, shared filtering and bounded streaming.
• Reuse cached residual-IP base LUTs across lists within the existing 8 MiB budget, preserving first-subtable offset rounding and error restoration. Avoid redundant PQ cosine query copies and RQ L2/cosine query-norm computation.
• Normalize cosine queries before probing/scanning (and before PQ OPQ). PQ cosine uses half the floating-point ADC squared-distance estimate; IP uses negative estimated dot products with residual coarse contributions included once. No cosine/IP partial-sum pruning is used.
• Add oracle, ULP/boundary, nonfinite-data, zero-vector, filter, statistics, cache/streaming and L2/top-K regression coverage, and update the Rust/docs capability and estimator descriptions.

Existing L2 and top-K behavior, the storage format and C/JNI APIs remain unchanged. No original-vector reranking or top-K fallback is introduced.

Validation

Run locally on macOS/AArch64:

• cargo test --offline --locked --workspace: 662 passed, 2 ignored after the performance fixes.

• cargo test --offline --locked --release --workspace -- --test-threads=1: 662 passed, 2 ignored after the performance fixes. This serializes the test harness, not the explicit 1/4-worker Rayon pools exercised by the tests.

• cargo clippy --offline --locked --workspace --all-targets -- -D warnings: passed.

• cargo fmt --all -- --check and git diff --check: passed.

• python3 tools/check_license_headers.py: passed.

• cargo test --offline --locked -p paimon-vindex-core --test range_search pq_range_coarse: both coarse-parallel regression tests passed.

• Local documentation links/anchors: 192 references passed across six changed HTML pages.

• PQ coverage includes all metrics, PQ4/PQ8, both residual modes, dense 64D OPQ, scalar-oracle comparisons, adjacent-ULP membership, single/batch/filter equivalence, 1/4 workers and optimize_for_search invariance. Streaming crosses the production 8 MiB LUT cache budget for PQ8; PQ4 scratch fallback is covered separately with a forced budget.

• Targeted performance check on Apple M2 Pro: a synthetic PQ8 thin-list fixture (d=768, m=96, 64 lists of four rows, zero coarse centroids, 16 queries, full probe, one worker) improved residual-IP batch median from 42.420 ms to 1.285 ms over five alternating before/after runs, with identical result checksums. This is a LUT-heavy fixture, not a general throughput guarantee.

• Additional residual-IP streaming coverage uses nonzero coarse centroids, 33 queries crossing the PQ8 cache budget, PQ4/PQ8, 1/4 workers, filters and early stopping on nonfinite estimates.

Local independent read-only agent reviews of shared semantics/Flat/SQ/RQ and the PQ extension found no remaining in-scope blockers. A follow-up read-only review of the rebase conflict resolution and retained upstream optimizations also passed. Independent review of the LUT/query-preparation fixes and added streaming regression also found no blockers. Maintainer review and hosted CI remain separate requirements before merge.

Known validation limitation

Default-concurrent Release test runs have intermittently failed unchanged top-K tests that assert work was observed on multiple Rayon workers. A clean archive of an earlier #107 baseline, compiled into a separate target directory, reproduced the same category of IVF-Flat worker-count assertion failure. Isolated tests and the complete serialized Release test harness pass. This is not a claim that default-concurrent Release is consistently green.

The expanded SQ range parallelism test uses a bounded worker handshake to avoid that scheduling race in its own coverage; unrelated top-K tests are not changed here.

Rollback / follow-up

No data migration or format change is involved. Reverting this single squashed commit leaves the merged #107 implementation intact. Before merging, inspect hosted CI and maintainer feedback.

[jerry-024]

+1