Skip to content

feat: IVF-RaBitQ index type (#218) - #235

Merged
s2x merged 2 commits into
mainfrom
feat/218-ivf-rabitq
Sep 29, 2026
Merged

s2x merged 2 commits into
mainfrom
feat/218-ivf-rabitq

Conversation

@s2x

@s2x s2x commented Sep 29, 2026

Copy link
Copy Markdown
Member

Closes #218.

Summary

IVF_RABITQ is a new index type in v0.7.0 — an IVF partitioned index storing RaBitQ-quantized vectors. zvec-php could neither create one nor send its query params.

ZVecIndexParams::forIvfRabitq(int $metricType, int $nList = 1024, int $totalBits = 7, int $sampleCount = 0)
ZVecVectorQuery::setIvfRabitqParams(int $nprobe = 10)

ZVec::INDEX_TYPE_IVF_RABITQ    = 7
ZVec::QUERY_PARAM_IVF_RABITQ   = 7

Both constants match the upstream enum and C ABI. Unlike VAMANA/DISKANN there is no swap to reconcile.

Design notes

No quantize type is passed. Upstream rejects an IVF index carrying QuantizeType::RABITQ with "use the dedicated IVF_RABITQ index instead", and IvfRabitqIndexParams always sets RABITQ itself. So the builder deliberately omits it.

totalBits is validated in PHP. The upstream core only logs an out-of-range total_bits at build time rather than raising, so a bad value would fail late and quietly. Range 1..9, rejected up front.

scale_factor is not exposed. engine_helper copies only nprobe for this index type, so a setter for it would silently do nothing.

Five load-bearing C++ changes beyond the two setters

  • to_index_type() / from_index_type() need case 7 in both directions. Without the latter, getFieldSchema()->getIndexType() reports 0 for the new index.
  • ensure_query_params_for_field() needs an IVF_RABITQ case. Without it, a radius-only query falls through to the HNSW fallback and upstream rejects the whole query with a params-type mismatch — the radius-only query ok line in the test pins exactly this down.
  • The group-by switch gets the same case, so the error message stays honest.
  • The legacy query() path needs case 7 in validate_query_param_type() and a branch in apply_query_params(). It has no dedicated nprobe argument, so it reuses the existing IVF one.

Platform, and it was actually exercised

Upstream supports RaBitQ on Linux x86_64 with AVX2+FMA or AVX-512 only, and rejects an FP64 field, a dimension outside 64–4095, or a metric other than L2/IP/COSINE. createIndex() fails with NOT_SUPPORTED elsewhere, so the functional test skips off that platform.

This machine is Linux x86_64 with both AVX2 and AVX-512, so the test ran rather than skipped:

index type: 7
queryVector top: 42
query top: 42
radius-only query ok
hnsw params rejected (code 3)
PASS: IVF-RaBitQ index works
dim 32 rejected (code 3)

A 64-document index over 128-dim vectors, with doc 42 retrieved for its own vector through both query paths.

One test-hygiene note

Building a RaBitQ index makes the bundled rabitqlib print FhtKacRotator is selected to stderr from a C++ static initialiser. It cannot be silenced from PHP and would otherwise land in the expected output, so the test uses --CAPTURE_STDIO STDOUT. Noted in a DESCRIPTION section so the flag does not look arbitrary.

Full suite:

Number of tests : 193               193
Tests skipped   :   0 (  0.0%)
Tests failed    :   0 (  0.0%)
Expected fail   :   2 (  1.0%)
Tests passed    : 191 ( 99.0%)

test_docs_consistency.phpt gained the new INDEX_TYPE_IVF_RABITQ row, and test_ffi_load.phpt the two new symbols. test_dbs/ is left with only .gitignore.

🤖 Generated with Claude Code

s2x added 2 commits September 29, 2026 08:25
IVF_RABITQ is a new index type in v0.7.0: an IVF partitioned index that
stores RaBitQ-quantized vectors. zvec-php could neither create one nor
send IVF_RABITQ query params.

- ZVecIndexParams::forIvfRabitq(metricType, nList, totalBits, sampleCount)
- ZVecVectorQuery::setIvfRabitqParams(nprobe)
- ZVec::INDEX_TYPE_IVF_RABITQ and ZVec::QUERY_PARAM_IVF_RABITQ, both 7,
  matching the upstream enum and C ABI. Unlike VAMANA/DISKANN there is no
  swap to reconcile.

Upstream also rejects an IVF index carrying QuantizeType::RABITQ with
"use the dedicated IVF_RABITQ index instead", so the builder deliberately
does not pass a quantize type: IvfRabitqIndexParams always sets RABITQ
itself.

PHP validates nList > 0, 1 <= totalBits <= 9 and sampleCount >= 0 before
any FFI call. total_bits matters here because the upstream core only logs
an out-of-range value at build time rather than raising, so a bad value
would fail late and quietly instead of being rejected.

scale_factor is not exposed on purpose: engine_helper copies only nprobe
for this index type, so a setter for it would silently do nothing.

C++ changes beyond the two setters, each of which is load-bearing:

- to_index_type() / from_index_type() need case 7 both ways. Without the
  latter, getFieldSchema()->getIndexType() reports 0 for the new index.
- ensure_query_params_for_field() needs an IVF_RABITQ case. Without it a
  radius-only query falls through to the HNSW fallback and upstream
  rejects the whole query with a params-type mismatch, which is what the
  radius-only test case pins down.
- The group-by switch gets the same case, so the error stays honest.
- The legacy query() path needs case 7 in validate_query_param_type() and
  a branch in apply_query_params(). It has no dedicated nprobe argument,
  so it reuses the existing IVF one.

Tests: 193/193, 0 skipped, 0 failed, 2 expected fail.
test_ivf_rabitq_index.phpt was run on this machine (Linux x86_64, AVX2 +
AVX-512, so not skipped) and passes: it creates a 64-document IVF_RaBitQ
index, gets 42 back for doc 42's vector through both queryVector() and the
legacy query(), and confirms the radius-only path, the params-type
mismatch and the dimension >= 64 rule. It skips off Linux x86_64, where
upstream rejects the index with NOT_SUPPORTED.

The index test uses --CAPTURE_STDIO STDOUT: building a RaBitQ index makes
the bundled rabitqlib print "FhtKacRotator is selected" to stderr from a C++
static initialiser. It cannot be silenced from PHP, and it would otherwise
land in the expected output.
@s2x
s2x merged commit 9fe8978 into main Sep 29, 2026
11 of 12 checks passed
@s2x
s2x deleted the feat/218-ivf-rabitq branch September 29, 2026 07:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: IVF RaBitQ index (IndexType::IVF_RABITQ = 7) with index and query params

1 participant