Repository navigation
feat(query): native hybrid multi-query (multiQuery()) (#217) - #241
Merged
Merged
Conversation
queryMulti() runs each sub-query separately through the legacy scalar path and fuses in PHP. That cannot do hybrid search at all: the scalar path knows nothing about an FTS clause, so an FTS sub-query -- which has no vector -- aborted on "Cannot instantiate FFI\CData of zero size". It also dropped everything living only in the C++ query handle (the FTS clause, DiskANN listSize, HNSW prefetch, setIncludeDocId, setOutputFields), and keyed results by field name, so two sub-queries on one field silently replaced the first. multiQuery() builds one upstream MultiQuery instead, so zvec runs the sub-queries in parallel and fuses in C++. queryMulti() is unchanged for everything it supported. It now rejects the two cases it could not handle, naming multiQuery() as the alternative: an FTS sub-query, and a duplicate field. Both used to fail with an unusable FFI error or silently return less than asked. The fused score is not comparable with queryMulti(). Upstream normalises weighted fusion per field with atan, not min-max, so the numbers differ from ZVecWeightedReRanker for identical inputs. RRF is the same 1/(k + rank + 1) either way, and the test asserts the exact fractions. This is precisely why queryMulti() keeps its PHP fusion. Weighted fusion is positional upstream and duplicate field names are legal, so a field-keyed weight map has no unambiguous meaning. It is rejected with an explicit message rather than silently mapped in the wrong order. The FFI surface uses a zvec_ffi_ prefix: upstream's own C API library exports zvec_multi_query_create and zvec_collection_multi_query with *different* signatures under the bare names, and identical names with different ABIs is a trap. It follows the existing zvec_ffi_initialize convention. A sub-query that only set radius/linear/refiner has no query_params_ yet, so the adapter builds the right ones for the field's index type at execute time; otherwise upstream rejects the whole multi-query with a params-type mismatch. The adapter keeps full copies of the sub-queries because those three values live on the holder, not on the target. Tests: 211/211, 0 skipped, 0 failed, 2 expected fail. - test_multi_query.phpt: FTS + dense fusion, the RRF fractions, filter, numCandidates, two sub-queries on the same field, weighted fusion, output fields, and five validation paths. - test_multi_query_bc.phpt: the two new queryMulti() rejections plus plain multi-vector and the native equivalent. - test_memory_multi_query.phpt: no handle or C string leak over 100 runs. - test_ffi_load.phpt gained the eleven new symbols. - test_multivector_query, test_multivector_weighted, test_rerankers, test_reranker_in_query and test_fts all pass unchanged. Returns plain ZVecDoc[] rather than ZVecRerankedDoc[]: the fusion happened upstream, so there are no per-field source ranks to report.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #217.
Summary
queryMulti()runs each sub-query separately through the legacy scalar path and fuses in PHP. That cannot do hybrid search at all — the scalar path knows nothing about an FTS clause, so an FTS sub-query (which has no vector) aborted onCannot instantiate FFI\CData of zero size. It also dropped everything living only in the C++ query handle, and keyed results by field name.multiQuery()builds one upstreamMultiQueryinstead, so zvec runs the sub-queries in parallel and fuses in C++:What the old path actually got wrong
VectorQueryHoldernever reachedzvec_collection_query_ex: the FTS clause, DiskANNlistSize, HNSW prefetch,setIncludeDocId(),setOutputFields().$queryResults[$vq->fieldName] = $docs.queryMulti()is unchanged for everything it supported, and now rejects the two cases it could not, namingmultiQuery()as the alternative:The scores are deliberately not comparable
Upstream normalises weighted fusion per field with
atan, not min-max, so the numbers differ fromZVecWeightedReRankerfor identical inputs. RRF is the same1/(k + rank + 1)in both. This is exactly whyqueryMulti()keeps its PHP fusion — changing it would have broken anyone relying on the values.The test asserts the RRF arithmetic rather than just the ordering, which is what pinned down the actual rank assignment (d1 and d2 are rank 0 in different per-field lists, so both score
1/61 + 1/62):Weighted fusion is positional
Upstream indexes weights by sub-query position, and duplicate field names are legal — so a field-keyed map has no unambiguous meaning. It is rejected explicitly rather than silently mapped in the wrong order:
The FFI prefix is deliberate
zvec_ffi_multi_query_create, notzvec_multi_query_create. Upstream's own C API library exports those bare names with different signatures, and identical names with different ABIs is a trap. This follows the existingzvec_ffi_initializeconvention.One non-obvious detail
A sub-query that only set
radius/linear/refinerhas noquery_params_yet, so the adapter builds the right ones for the field's index type at execute time — otherwise upstream rejects the whole multi-query with a params-type mismatch. That is whyMultiQueryHolderkeeps full copies of the sub-queries rather than just theirQueryTarget: those three values live on the holder.Verification
test_multivector_query,test_multivector_weighted,test_rerankers,test_reranker_in_queryandtest_ftsall pass unchanged — the BC guarantee is tested, not just asserted.Returns plain
ZVecDoc[]rather thanZVecRerankedDoc[]: the fusion happened upstream, so there are no per-field source ranks to report.Out of scope per the issue: sparse sub-queries (needs the sparse query-vector setter, #210),
fromId()sub-queries,CallbackParams(a PHP callback invoked from C++), and group-by multi-query (upstream has none).🤖 Generated with Claude Code