diff --git a/CHANGELOG.md b/CHANGELOG.md index a687412..d3b93d0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -27,6 +27,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - A mismatched index type is left to upstream, which reports it at `ZVec::create()` time naming the field, so there is one source of truth rather than two. - FFI: `zvec_schema_add_field_string_with_index()`. The index params are cloned by upstream's `FieldSchema` constructor, so the `ZVecIndexParams` object may be freed as usual. - Test: `tests/test_schema_fts_index.phpt` (query without `createIndex()`, introspection before and after reopen, invert via params, the two validation errors, and backward compatibility of the existing arguments). + - **`ZVec::init()` jieba and FTS tuning options** (#221) - `?string $jiebaDictDir` sets the folder holding `jieba.dict.utf8` and `hmm_model.utf8` for the jieba FTS tokenizer, and `?float $ftsBruteForceByKeysRatio` (0.0–1.0) the point at which an FTS query stops walking posting lists and scores candidates one by one. Upstream default is 0.05. Both take effect only on the first successful `init()` in a process, like every other option. - Read back with `ZVec::getJiebaDictDir()` and `ZVec::getFtsBruteForceByKeysRatio()`. With no option given, the jieba dictionary shipped in `zvec_data/jieba_dict` is still picked up automatically. @@ -43,6 +44,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - FFI: `zvec_index_params_set_ivf_rabitq()`, `zvec_vector_query_set_ivf_rabitq_nprobe()`, the index type in `to_index_type()` / `from_index_type()`, an `IVF_RABITQ` case in `ensure_query_params_for_field()` and the group-by switch, and a `case 7` in the legacy `query()` path's `validate_query_param_type()` and `apply_query_params()`. The legacy path reuses the existing `ivfNprobe` argument, having no separate one. - `scale_factor` is deliberately not exposed: the upstream engine copies only `nprobe` for this index type, so exposing it would be misleading. - Tests: `tests/test_ivf_rabitq_params.phpt` (constants, validation, query-param plumbing) and `tests/test_ivf_rabitq_index.phpt` (real index, both query paths, radius-only through `ensure_query_params_for_field()`, a params-type mismatch, and the dimension rule). + - **DiskANN I/O backend introspection** (#224) - `ZVec::getIoBackendType()`, `ZVec::getIoBackendTypeName(int $type)` and `ZVec::getIoBackendDescription()` report the I/O backend zvec picked for DiskANN disk reads. Linux tries `io_uring`, then `libaio`, and falls back to synchronous `pread()`; macOS always uses `pread()`. The value matters a lot for DiskANN throughput and previously could not be observed at all. - The choice is process-wide and resolved lazily, so the getters work without `ZVec::init()`. New constants `ZVec::IO_BACKEND_PREAD`, `IO_BACKEND_LIBAIO` and `IO_BACKEND_IO_URING` (`0`/`1`/`2`), matching the upstream C ABI. @@ -98,6 +100,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed +- **The documented test command silently skipped most of the suite** (#215, #188) + - `php run-tests.php -n tests/` was documented as *the* way to run the suite, with the rationale that `-n` keeps a pre-installed legacy `zvec` extension from shadowing the FFI classes. That is only half the requirement. `-n` also strips `php.ini`, so wherever FFI is provided by a `conf.d` ini rather than compiled in, FFI disappears with it and **every** test reports `SKIP — reason: FFI extension not available`. + - Measured on a glibc host with PHP 8.5.4: 16 passed, **172 skipped**, 1 failed — and the run still printed a green-looking summary. A large skip count is a broken run, not a pass. + - The suite now needs both halves: `php run-tests.php -n -d extension=ffi.so tests/`. `-n` disables the legacy extension, `-d extension=ffi.so` keeps FFI available afterwards. If your PHP has FFI compiled in, drop the `-d` flag. + - `AGENTS.md` and `build_zvec.sh` updated, with a note that `Tests skipped` must be `0` before a run is trusted. + - `.github/scripts/run-tests.sh` already did the right thing and needed no change; the problem was the documentation only. + - **`fetch_zvec_sdk.sh` picked the musl SDK on glibc hosts** (#215) - The musl check was `ls /lib/ld-musl-*.so.1`, which also succeeds on a glibc machine that merely has `musl-tools` installed. The musl SDK was then unpacked and every test failed at `ZVec::ffi()` with `libc.musl-x86_64.so.1: cannot open shared object file`. - The detection now inspects the loader of the binary that will actually load the SDK (`ldd $(command -v php)`), falling back to `/bin/sh`, and assumes glibc when neither marker is present. Override the probe with `PHP_BINARY`. @@ -105,6 +114,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - **Crash at exit and lost `ZVec::init()` settings with a shared libzvec** (#215) - libzvec exports the `GlobalConfig` singleton but keeps its init guard local, so calling the inline `GlobalConfig::Instance()` from our module built a second copy: `init()` settings were silently reset and the object was destroyed twice at exit (`free(): chunks in smallbin corrupted`, SIGABRT). The FFI adapter and the extension now resolve the instance from libzvec via `dlsym`. Test: `tests/bug_0056.phpt`. - Release builds of the Linux FFI adapter embed libstdc++; its symbols are now kept private (`-Wl,--exclude-libs,ALL`). Before, the dynamic linker bound part of them to the system libstdc++ that PHP already loads through libxml2/libicu, and `schema()` segfaulted inside `IndexParams::to_string()`. + - **`run-tests.php` still overwrote hand-written `tests/.php` files** (#187) — the temp copy of the extracted test used the old path; it now uses the `.php.tmp-extract` suffix too. - **`ZVecFieldSchema::getIndexType()` swapped VAMANA and DISKANN** — the C++ enum order differs from the PHP constants; the value is now mapped back explicitly. Test: `tests/bug_0055.phpt`. @@ -129,23 +139,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Note: a second `set*Params()` call no longer implicitly resets previously set radius/linear/refiner (merge semantics — set `setRadius(0.0)` to restore the default). - Added regression test `test_query_params_order.phpt`. -### Fixed - - **Random rotation for INT8/INT4 quantization** (#177) - `ZVecIndexParams::setQuantizerEnableRotate(bool)` (fluent) enables random rotation before INT8/INT4 quantization for HNSW, Flat, IVF, and Vamana indexes — reduces quantization error and improves recall on quantized indexes. - Mirrors upstream zvec v0.6.0 `QuantizerParam(enable_rotate)` (C API: `zvec_index_params_set_quantizer_enable_rotate`). -### Fixed - - **`queryVector()` with Vamana / HNSW RaBitQ query params** (#193) - `ZVecVectorQuery::setVamanaParams()` and `setHnswRabitqParams()` previously created an `HnswQueryParams` object, so `queryVector()` on Vamana/RaBitQ indexes was always rejected by the engine (`query params type does not match the index type of vector field`). - FFI: added `zvec_vector_query_set_vamana_ef_search()` (`VamanaQueryParams`) and `zvec_vector_query_set_hnsw_rabitq_ef()` (`HnswRabitqQueryParams`), mirroring `apply_query_params`; the legacy `query()` path was unaffected. - Added regression tests `test_vector_query_vamana_params.phpt` and `test_vector_query_rabitq_params.phpt` (RaBitQ runs on Linux x86_64 only). -### Fixed - - **Deprecated index creation warnings** (#169) - The FFI bindings’ `createHnswIndex()`, `createHnswRabitqIndex()`, `createFlatIndex()`, and `createIvfIndex()` now emit `E_USER_DEPRECATED` before delegating to the unified `createIndex()` API. + - **Regression gate against zvec v0.6.0** (#175) - Full `.phpt` suite verified (FFI mode, `-n`): 0 failures, only expected XFAILs (VECTOR_FP64, upstream-blocked) and the expected RaBitQ platform SKIP (Linux x86_64 only) - All 12 legacy `tests/bug_*.php` scripts pass against v0.6.0; cleaned-up statuses for `bug_0005_cleanup_after_failed_ops.php` and `bug_0006_rocksdb_lock.php` (no issues observed on v0.6.0) @@ -153,6 +158,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Legacy `test_installer_platform.php` accepts the `macOS` label prefix on Darwin, matching its `.phpt` counterpart - Documented the `-n` requirement (pre-installed legacy extension shadows FFI classes — #188) and the `run-tests.php` legacy-file deletion hazard (#187) in `AGENTS.md` / `README.md` +### Known issues + +- **Cross-module `operator new` / `operator delete` mismatch with the prebuilt SDK** (#228) + - valgrind reports `Mismatched free() / delete` where objects allocated by `operator new` inside `libzvec.so` are released by `operator delete(void*, unsigned long)` in `libzvec_ffi.so` (seen in `zvec_collection_insert` and `zvec_schema_free`). The SDK carries its `operator new` / `operator delete` as local symbols, so it uses a different allocation routine than the adapter, which shares the dynamic `libstdc++` with PHP. + - This is **not** the cause of the exit-time crash fixed above; that one was the duplicate `GlobalConfig` singleton. This one is currently benign — both paths end in the same glibc `free()` on a glibc-allocated chunk, and the full suite passes — but it is undefined behaviour and could corrupt the heap if the two implementations ever diverge (hardened allocators, jemalloc/tcmalloc preloaded under PHP). + - Not yet fixed. Candidate approaches are listed in #228; the preferred one is to make the adapter match the SDK's allocation routine. Verified on Linux x86_64 with the glibc SDK; not reproducible against a source build. + ## [0.6.0] - 2026-07-30 Adapter-only release: the zvec dependency moved from v0.5.x to v0.6.0. **The diff --git a/MIGRATION.md b/MIGRATION.md index 8b6b667..dedef1f 100644 --- a/MIGRATION.md +++ b/MIGRATION.md @@ -1,3 +1,12 @@ +# Migration Guide + +Two guides live in this file, each covering one upgrade: + +- **[v0.6.0 → v0.7.0](#migration-guide-v060--v070)** (at the bottom) — new + additive API, the prebuilt zvec SDK, and the test-suite invocation change. +- **[v0.4.x → v0.5.0](#migration-guide-v04x--v050)** (below) — replacing the + deprecated v0.4.x APIs with the modern ones. + # Migration Guide: v0.4.x → v0.5.0 This guide helps users of the deprecated v0.4.x APIs migrate to the modern @@ -435,3 +444,195 @@ creation fails at runtime with `DiskAnn factory entries are not registered`. `core_framework` and `core_knn_cluster` are deliberately *not* listed: `libzvec_core.a` already contains those objects and naming them duplicates every shared symbol. + +--- + +# Migration Guide: v0.6.0 → v0.7.0 + +Everything below is **additive**. No existing method was removed, renamed or +retyped, so existing code keeps working unchanged. The one change that can +affect you is how the test suite is invoked (see the last section), and the +switch to a prebuilt zvec SDK. + +## New: a scalar index can be declared in the schema + +Previously an FTS or invert index on a STRING field needed two steps: create +the collection, then call `createIndex()`. The index can now be part of the +schema, so it exists from the first insert. + +```php +// BEFORE — two steps, index absent until createIndex(): +$schema = new ZVecSchema('docs'); +$schema->addString('body'); +$collection = ZVec::create($path, $schema); +$collection->createIndex('body', ZVecIndexParams::forFts()); + +// AFTER — one step: +$schema = new ZVecSchema('docs'); +$schema->addString('body', indexParams: ZVecIndexParams::forFts()); +$collection = ZVec::create($path, $schema); +``` + +The new argument is last and optional, so `addString('body')` and +`addString('body', withInvertIndex: true)` are unchanged. Two notes: + +- Passing both `$withInvertIndex` and `$indexParams` now throws + (`Use either $withInvertIndex or $indexParams, not both`) instead of one + silently winning. +- A mismatched index type is reported by `ZVec::create()` naming the field, e.g. + `scalar field[body] does not support vector index params`. That is unchanged + behaviour, just now reachable without a second call. + +## New: IVF-RaBitQ index type + +An IVF partitioned index storing RaBitQ-quantized vectors. Mirrors the Python +`IvfRabitqIndexParam` / `IvfRabitqQueryParam`. + +```php +$collection->createIndex('v', ZVecIndexParams::forIvfRabitq( + metricType: ZVecSchema::METRIC_L2, + nList: 1024, + totalBits: 7, // 1..9 + sampleCount: 0, +)); + +$query = (new ZVecVectorQuery('v', $vector)) + ->setTopk(10) + ->setIvfRabitqParams(nprobe: 16); +``` + +New constants `ZVec::INDEX_TYPE_IVF_RABITQ` and `ZVec::QUERY_PARAM_IVF_RABITQ`, +both `7`. + +**Platform:** upstream supports RaBitQ on **Linux x86_64 with AVX2+FMA or +AVX-512 only**, and rejects an FP64 field, a dimension outside 64–4095, or a +metric other than L2/IP/COSINE. On other platforms `createIndex()` fails with +`NOT_SUPPORTED`. Note that `QUANTIZE_RABITQ` is *not* what selects this index — +it is implied by the index type, and passing it to plain `forIvf()` is rejected. + +## New: Vamana `two_pass_build` and query prefetch + +`two_pass_build` runs a second full-graph Vamana construction pass: better graph +quality, slower build. It is a trailing optional parameter, so existing +positional calls to `forVamana()` are unaffected. + +```php +$params = ZVecIndexParams::forVamana( + metricType: ZVecSchema::METRIC_IP, + maxDegree: 64, + // ... + twoPassBuild: true, +); +``` + +`setVamanaPrefetch()` tunes search-time software prefetch, the counterpart of +the existing `setHnswPrefetch()`. A `prefetchOffset` of `0` disables prefetch; +`0` lines means "derive from vector size". The call order relative to +`setVamanaParams()` does not matter. + +```php +$query = (new ZVecVectorQuery('v', $vector)) + ->setTopk(10) + ->setVamanaPrefetch(256, 4); +``` + +> **Known limitation:** prefetch — HNSW *and* Vamana — is honoured only by +> `queryVector()`. The legacy `query()` path sends just `queryParamType`, +> `ef`/`nprobe`, `radius`, `isLinear` and `isUsingRefiner` upstream, so prefetch +> is silently dropped there. This is not new in v0.7.0; it applies equally to +> `setHnswPrefetch()`. + +## New: `ZVec::init()` tuning options + +```php +ZVec::init( + // ...existing parameters... + ?string $jiebaDictDir = null, // folder with jieba.dict.utf8 + hmm_model.utf8 + ?float $ftsBruteForceByKeysRatio = null, // 0.0-1.0, upstream default 0.05 +); +``` + +Read back with `ZVec::getJiebaDictDir()` and +`ZVec::getFtsBruteForceByKeysRatio()`. With neither option given, the jieba +dictionary shipped in `zvec_data/jieba_dict` is still found automatically. + +Like every other `init()` option, both take effect only on the **first** +successful `init()` in a process; upstream ignores later calls. `null` is the +"keep the upstream default" marker, because `0.0` is a real value upstream +accepts. + +Both are validated in PHP before any FFI call, which is the only place the +check can live: upstream `GlobalConfig::initialize()` sets its initialized flag +*before* validating, so a rejected value would leave the library marked +initialized and make every later `init()` a silent no-op. A `jiebaDictDir` +missing its dictionary files is rejected too, for a worse reason — cppjieba +calls `abort()`, killing the process with exit code 134 rather than raising a +catchable error. + +## New: DiskANN I/O backend introspection + +zvec picks an I/O backend for DiskANN disk reads on first use — on Linux it +tries `io_uring`, then `libaio`, then falls back to synchronous `pread()`; +macOS always uses `pread()`. The choice dominates DiskANN throughput and was +previously unobservable. + +```php +ZVec::getIoBackendType(); // ZVec::IO_BACKEND_PREAD | LIBAIO | IO_URING +ZVec::getIoBackendTypeName(2); // "io_uring" +ZVec::getIoBackendDescription(); // human-readable, with install hints on Linux +``` + +New constants `ZVec::IO_BACKEND_PREAD` / `IO_BACKEND_LIBAIO` / +`IO_BACKEND_IO_URING` (`0` / `1` / `2`). The getters work without +`ZVec::init()` — the backend is resolved lazily and the value is process-wide. + +If you see `pread` on Linux, neither io_uring nor libaio could be loaded, so +DiskANN reads are synchronous. Installing libaio (`libaio1t64` on Ubuntu +24.04+) enables async I/O. + +## Changed: the test suite must be run with FFI enabled + +This is the one change that can break someone's workflow. + +```bash +# BEFORE — silently skipped most of the suite on many machines: +php run-tests.php -n tests/ + +# AFTER: +php run-tests.php -n -d extension=ffi.so tests/ +``` + +`-n` keeps a pre-installed legacy `zvec` extension from shadowing the FFI +classes, but it also strips `php.ini` — so wherever FFI is provided by a +`conf.d` ini rather than compiled in, FFI disappears with it. Measured on a +glibc host with PHP 8.5.4: 16 passed, **172 skipped**, 1 failed, and the run +still printed a green-looking summary. + +If your PHP has FFI compiled in, drop the `-d extension=ffi.so` flag. Either +way, check that `Tests skipped` is `0` before trusting a run — a large skip +count means FFI was not loaded, not that the code is fine. + +## Changed: zvec is now a prebuilt SDK + +zvec is no longer compiled from source. `./fetch_zvec_sdk.sh` downloads the +official prebuilt SDK from the +[alibaba/zvec releases](https://github.com/alibaba/zvec/releases) (pinned +SHA-256) and `./build_zvec.sh` builds only our adapter. + +If you build from source or vendor `libzvec`, the important consequence is +that `libzvec` and `zvec_data/` (the jieba dictionary) must sit in the same +directory as `libzvec_ffi`. The adapter resolves libzvec via an `$ORIGIN` / +`@loader_path` rpath, so a Composer install under `lib/` needs all three. + +Release asset names changed: `libzvec_ffi-.tar.gz` instead of +`libzvec_ffi-ubuntu24-x86_64.tar.gz`. `vendor/bin/zvec-install` handles this +automatically; only custom CI that fetches assets by name needs updating. + +## Unchanged + +- Every pre-v0.7.0 method, constant and behaviour. +- All previously deprecated `create*Index()` and `addField*()` methods still + work and still emit `E_USER_DEPRECATED`. +- `queryById()`, `queryMulti()`, `queryWithReranker()` and + `groupByQuery()` signatures. + diff --git a/README.md b/README.md index 9afc482..c653292 100644 --- a/README.md +++ b/README.md @@ -6,6 +6,13 @@ Two binding modes are available: - **Native PHP extension** (`php-ext/`) — compiled C++ extension, no FFI overhead (recommended) - **FFI bindings** (`php/` + `ffi/`) — pure PHP via FFI, no compilation needed beyond the shared library +**Bundled zvec version: `v0.7.0`.** zvec itself is not compiled from source — we +link against the official prebuilt SDK published by +[alibaba/zvec](https://github.com/alibaba/zvec/releases) and compile only our +adapter. See [MIGRATION.md](MIGRATION.md#migration-guide-v060--v070) for what +changed in this version, and note the one thing that will break your workflow: +the test-suite command now needs an extra flag. + ## Overview zvec-php provides PHP bindings for the zvec vector database through FFI (Foreign Function Interface). It allows you to create collections, insert documents with vectors, perform similarity searches, and manage indexes - all from PHP. @@ -46,7 +53,7 @@ in `lib/`: our small FFI adapter (`libzvec_ffi`, from this project's GitHub Rele and the official prebuilt zvec SDK library (`libzvec`, from [alibaba/zvec releases](https://github.com/alibaba/zvec/releases), pinned SHA-256). The version is auto-detected from Composer's `installed.json`. If omitted, -specify it explicitly: `vendor/bin/zvec-install v0.4.10`. +specify it explicitly: `vendor/bin/zvec-install v0.7.0`. Supported platforms (all pre-built): - **Linux x86_64 / aarch64 (glibc 2.28+)** — `libzvec_ffi-linux-{x86_64,aarch64}.tar.gz` @@ -77,11 +84,14 @@ Requires CMake 3.14+ and a C++17 compiler. zvec itself is no longer built from s ### 3. Verify installation (FFI mode) ```bash -php run-tests.php -n tests/ +php run-tests.php -n -d extension=ffi.so tests/ ``` -> **Note:** The `-n` flag (no php.ini) is required to avoid a legacy -> pre-installed `zvec` PHP extension shadowing the FFI classes (#188). +> **Note:** The goal is FFI enabled *and* the legacy `zvec` extension +> disabled. `-n` (no php.ini) handles the second, but it also strips any +> conf.d ini that provides FFI — on such a machine the suite then reports every +> test as SKIP. `-d extension=ffi.so` restores FFI afterwards. Check that +> `Tests skipped` is `0`; a run with skips is a broken run, not a pass (#188). ### Alternative: Build the native PHP extension @@ -829,15 +839,17 @@ Returned by the `set*Params()` / `setFts()` methods on `ZVecVectorQuery` via its ### .phpt test suite (FFI mode) ```bash -php run-tests.php -n tests/ +php run-tests.php -n -d extension=ffi.so tests/ ``` -> **Note:** Run with `-n` (no php.ini). A legacy pre-installed `zvec` PHP -> extension (v0.4.10) shadows the FFI classes and breaks the suite (#188). +> **Note:** `-n` disables php.ini, which avoids a legacy pre-installed `zvec` +> extension shadowing the FFI classes (#188) but can also take FFI with it. If +> your PHP has FFI compiled in, plain `-n` is enough; otherwise add +> `-d extension=ffi.so`. Either way `Tests skipped` must be `0`. ### Integration tests ```bash -php run-tests.php -n tests/ +php run-tests.php -n -d extension=ffi.so tests/ ``` ## Project Structure diff --git a/tests/test_docs_consistency.phpt b/tests/test_docs_consistency.phpt index dc872f2..f431281 100644 --- a/tests/test_docs_consistency.phpt +++ b/tests/test_docs_consistency.phpt @@ -152,6 +152,35 @@ if (!str_contains($mig, 'backward')) { $failures++; } +// 3b. MIGRATION.md carries a v0.6.0 -> v0.7.0 section that states the new API +// is additive, so a reader can tell whether their code breaks. +echo "v0.7.0 section: " . (str_contains($mig, 'Migration Guide: v0.6.0 → v0.7.0') ? "present" : "MISSING") . "\n"; +if (!str_contains($mig, 'Migration Guide: v0.6.0 → v0.7.0')) { + $failures++; +} +echo "v0.7.0 states additive: " . (str_contains($mig, 'additive') ? "yes" : "NO") . "\n"; +if (!str_contains($mig, 'additive')) { + $failures++; +} +// Each new public API added in v0.7.0 must appear in the migration guide, so +// the guide does not silently fall behind the README. +foreach ([ + 'forIvfRabitq', + 'setIvfRabitqParams', + 'twoPassBuild', + 'setVamanaPrefetch', + 'jiebaDictDir', + 'ftsBruteForceByKeysRatio', + 'getIoBackendType', + 'indexParams', +] as $api) { + $found = str_contains($mig, $api); + echo str_pad($api, 26) . ' ' . ($found ? 'documented' : 'MISSING FROM MIGRATION') . "\n"; + if (!$found) { + $failures++; + } +} + // 4. The zvec SDK version pin is v0.7.0 everywhere it is declared. foreach (['build_zvec.sh', 'fetch_zvec_sdk.sh', 'src/Installer.php'] as $f) { $c = file_get_contents("$root/$f"); @@ -190,6 +219,16 @@ setHnswPrefetch 1 setIncludeDocId 1 migration section: present states BC: yes +v0.7.0 section: present +v0.7.0 states additive: yes +forIvfRabitq documented +setIvfRabitqParams documented +twoPassBuild documented +setVamanaPrefetch documented +jiebaDictDir documented +ftsBruteForceByKeysRatio documented +getIoBackendType documented +indexParams documented build_zvec.sh v0.7.0 fetch_zvec_sdk.sh v0.7.0 src/Installer.php v0.7.0