Skip to content

feat: add range search across language bindings - #110

Merged
jerry-024 merged 5 commits into
apache:mainfrom
JunRuiLee:feat/range-search-bindings
Sep 22, 2026
Merged

jerry-024 merged 5 commits into
apache:mainfrom
JunRuiLee:feat/range-search-bindings

Conversation

@JunRuiLee

Copy link
Copy Markdown
Contributor

Summary

Expose the core range-search functionality from #108 through the existing C ABI/generated header, C++, JNI/Java, and Python bindings.

  • Support IVF-Flat, IVF-SQ, IVF-PQ, and IVF-RQ with L2, cosine, and inner product, for single/batch queries with or without a serialized Roaring allow-list.
  • Delegate raw half-open distance bands and public endpoint conversion to core, including structural unboundedness, strict/inclusive boundaries, and inner-product direction reversal.
  • Return variable-length CSR results with per-query statistics, call-level list reads, and reader capability queries.
  • Add explicit native-result ownership, exception-safe cleanup, checked lengths/conversions, and validation/error propagation.
  • Preserve existing top-K APIs and behavior. Python still imports and executes top-K against older/partial native libraries; unavailable range APIs report a clear upgrade error.

Tests and documentation

  • Add a core-only oracle generator and 168 shared cases consumed independently by C, C++, Java, and Python. Cases cover the full family/metric/API matrix, bounded/unbounded/empty bands, and absent/non-empty/empty filters.
  • Compare per-query label/distance-bit multisets, CSR offsets, and all counters without imposing an ordering guarantee absent from core.
  • Add boundary, overflow, malformed-input, ownership, callback/reentry, old-library compatibility, and unaligned-NumPy-input regression tests.
  • Document each language API and ownership contract; wire oracle generation and consumption into all four binding CI jobs.

Validation

Validated locally on macOS:

  • cargo fmt --all -- --check
  • cargo clippy --all-targets --workspace -- -D warnings
  • cargo test --workspace: 676 passed, 2 ignored
  • C11/C++17 strict builds (-Wall -Wextra -Werror), existing and new binding tests, and ASan/UBSan runs: passed; 168 oracle cases per binding
  • Java Maven API tests, native validation/panic-boundary/handle-safety tests, and the range oracle under -Xcheck:jni: passed; 168 oracle cases
  • Python tests with PVI_RANGE_FIXTURES enabled: 497 passed, including 168 oracle cases
  • ASF license-header checks and git diff --check: passed

Independent pre-PR reviews were completed; the older-native-library and unaligned-input findings were fixed and re-reviewed. C++/JNI allocation-failure paths were reviewed but are not exhaustively fault-injected. Cross-platform CI remains to run on this PR.

Scope and compatibility

Based on main at 0e25175 (#108). No core source/tests, storage-format, algorithm, or performance-tuning changes are included. DiskANN remains unsupported for range search. No data migration is required; rollback is a revert of this PR.

The range oracle tests use fminf/fmaxf, which require an explicit libm dependency when linking on Linux. Link the C test executable against m on UNIX without changing the binding or core behavior.

Validated with CMake, -Wall -Wextra -Werror, all C tests including 168 core oracle cases, cargo fmt, and the ASF header check. Linux validation follows in PR CI.
Comment thread java/src/main/java/org/apache/paimon/index/vector/VectorRangeSearchResult.java Outdated
Comment thread include/paimon_vindex.hpp
Add indexed Java result access, bound JNI metadata staging, and cover memory and ownership regressions.
return labels[hitIndex];
}

public float[] distances() {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Major] Make raw distance semantics explicit across bindings

fromEndpoints accepts public-space values, but range results expose core values as plain distances (L2 is squared and inner product is negated). A caller that compares a returned value with the original endpoint therefore gets the wrong answer. Please apply one explicit contract across Rust, C, C++, Java, and Python: name result values rawDistances/raw_distances (including indexed accessors), and make raw-band constructors internal or explicitly raw-named so the public construction path is fromEndpoints.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, fixed in 3cdd045. Unified raw_distances/rawDistances (including indexed/query access) across all five languages, with explicit raw constructors and public endpoint factories. Added API-contract/endpoint regressions and updated docs; numeric values and zero-copy behavior are unchanged.

Use raw-distance names across Rust, C, C++, Java, and Python, including indexed and per-query access. Make raw band construction explicit and keep public endpoint conversion as the normal entry point.

Document the range API migration and add contract, endpoint, ABI-layout, and zero-copy regressions. Preserve result values, ownership, C ABI layout, and existing top-K APIs.

@jerry-024 jerry-024 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@jerry-024
jerry-024 merged commit 17b1941 into apache:main Sep 22, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants