feat: IVF-RaBitQ index type (#218) - #235
Merged
Merged
Conversation
IVF_RABITQ is a new index type in v0.7.0: an IVF partitioned index that stores RaBitQ-quantized vectors. zvec-php could neither create one nor send IVF_RABITQ query params. - ZVecIndexParams::forIvfRabitq(metricType, nList, totalBits, sampleCount) - ZVecVectorQuery::setIvfRabitqParams(nprobe) - ZVec::INDEX_TYPE_IVF_RABITQ and ZVec::QUERY_PARAM_IVF_RABITQ, both 7, matching the upstream enum and C ABI. Unlike VAMANA/DISKANN there is no swap to reconcile. Upstream also rejects an IVF index carrying QuantizeType::RABITQ with "use the dedicated IVF_RABITQ index instead", so the builder deliberately does not pass a quantize type: IvfRabitqIndexParams always sets RABITQ itself. PHP validates nList > 0, 1 <= totalBits <= 9 and sampleCount >= 0 before any FFI call. total_bits matters here because the upstream core only logs an out-of-range value at build time rather than raising, so a bad value would fail late and quietly instead of being rejected. scale_factor is not exposed on purpose: engine_helper copies only nprobe for this index type, so a setter for it would silently do nothing. C++ changes beyond the two setters, each of which is load-bearing: - to_index_type() / from_index_type() need case 7 both ways. Without the latter, getFieldSchema()->getIndexType() reports 0 for the new index. - ensure_query_params_for_field() needs an IVF_RABITQ case. Without it a radius-only query falls through to the HNSW fallback and upstream rejects the whole query with a params-type mismatch, which is what the radius-only test case pins down. - The group-by switch gets the same case, so the error stays honest. - The legacy query() path needs case 7 in validate_query_param_type() and a branch in apply_query_params(). It has no dedicated nprobe argument, so it reuses the existing IVF one. Tests: 193/193, 0 skipped, 0 failed, 2 expected fail. test_ivf_rabitq_index.phpt was run on this machine (Linux x86_64, AVX2 + AVX-512, so not skipped) and passes: it creates a 64-document IVF_RaBitQ index, gets 42 back for doc 42's vector through both queryVector() and the legacy query(), and confirms the radius-only path, the params-type mismatch and the dimension >= 64 rule. It skips off Linux x86_64, where upstream rejects the index with NOT_SUPPORTED. The index test uses --CAPTURE_STDIO STDOUT: building a RaBitQ index makes the bundled rabitqlib print "FhtKacRotator is selected" to stderr from a C++ static initialiser. It cannot be silenced from PHP, and it would otherwise land in the expected output.
This was referenced Sep 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #218.
Summary
IVF_RABITQis a new index type in v0.7.0 — an IVF partitioned index storing RaBitQ-quantized vectors. zvec-php could neither create one nor send its query params.Both constants match the upstream enum and C ABI. Unlike
VAMANA/DISKANNthere is no swap to reconcile.Design notes
No quantize type is passed. Upstream rejects an IVF index carrying
QuantizeType::RABITQwith "use the dedicated IVF_RABITQ index instead", andIvfRabitqIndexParamsalways setsRABITQitself. So the builder deliberately omits it.totalBitsis validated in PHP. The upstream core only logs an out-of-rangetotal_bitsat build time rather than raising, so a bad value would fail late and quietly. Range1..9, rejected up front.scale_factoris not exposed.engine_helpercopies onlynprobefor this index type, so a setter for it would silently do nothing.Five load-bearing C++ changes beyond the two setters
to_index_type()/from_index_type()needcase 7in both directions. Without the latter,getFieldSchema()->getIndexType()reports0for the new index.ensure_query_params_for_field()needs anIVF_RABITQcase. Without it, a radius-only query falls through to the HNSW fallback and upstream rejects the whole query with a params-type mismatch — theradius-only query okline in the test pins exactly this down.query()path needscase 7invalidate_query_param_type()and a branch inapply_query_params(). It has no dedicated nprobe argument, so it reuses the existing IVF one.Platform, and it was actually exercised
Upstream supports RaBitQ on Linux x86_64 with AVX2+FMA or AVX-512 only, and rejects an FP64 field, a dimension outside 64–4095, or a metric other than L2/IP/COSINE.
createIndex()fails withNOT_SUPPORTEDelsewhere, so the functional test skips off that platform.This machine is Linux x86_64 with both AVX2 and AVX-512, so the test ran rather than skipped:
A 64-document index over 128-dim vectors, with doc 42 retrieved for its own vector through both query paths.
One test-hygiene note
Building a RaBitQ index makes the bundled
rabitqlibprintFhtKacRotator is selectedto stderr from a C++ static initialiser. It cannot be silenced from PHP and would otherwise land in the expected output, so the test uses--CAPTURE_STDIO STDOUT. Noted in aDESCRIPTIONsection so the flag does not look arbitrary.Full suite:
test_docs_consistency.phptgained the newINDEX_TYPE_IVF_RABITQrow, andtest_ffi_load.phptthe two new symbols.test_dbs/is left with only.gitignore.🤖 Generated with Claude Code