crazy-goat · s2x · Sep 29, 2026 · Sep 29, 2026
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -27,6 +27,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   - A mismatched index type is left to upstream, which reports it at `ZVec::create()` time naming the field, so there is one source of truth rather than two.
   - FFI: `zvec_schema_add_field_string_with_index()`. The index params are cloned by upstream's `FieldSchema` constructor, so the `ZVecIndexParams` object may be freed as usual.
   - Test: `tests/test_schema_fts_index.phpt` (query without `createIndex()`, introspection before and after reopen, invert via params, the two validation errors, and backward compatibility of the existing arguments).
+
 - **`ZVec::init()` jieba and FTS tuning options** (#221)
   - `?string $jiebaDictDir` sets the folder holding `jieba.dict.utf8` and `hmm_model.utf8` for the jieba FTS tokenizer, and `?float $ftsBruteForceByKeysRatio` (0.0–1.0) the point at which an FTS query stops walking posting lists and scores candidates one by one. Upstream default is 0.05. Both take effect only on the first successful `init()` in a process, like every other option.
   - Read back with `ZVec::getJiebaDictDir()` and `ZVec::getFtsBruteForceByKeysRatio()`. With no option given, the jieba dictionary shipped in `zvec_data/jieba_dict` is still picked up automatically.
@@ -43,6 +44,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   - FFI: `zvec_index_params_set_ivf_rabitq()`, `zvec_vector_query_set_ivf_rabitq_nprobe()`, the index type in `to_index_type()` / `from_index_type()`, an `IVF_RABITQ` case in `ensure_query_params_for_field()` and the group-by switch, and a `case 7` in the legacy `query()` path's `validate_query_param_type()` and `apply_query_params()`. The legacy path reuses the existing `ivfNprobe` argument, having no separate one.
   - `scale_factor` is deliberately not exposed: the upstream engine copies only `nprobe` for this index type, so exposing it would be misleading.
   - Tests: `tests/test_ivf_rabitq_params.phpt` (constants, validation, query-param plumbing) and `tests/test_ivf_rabitq_index.phpt` (real index, both query paths, radius-only through `ensure_query_params_for_field()`, a params-type mismatch, and the dimension rule).
+
 - **DiskANN I/O backend introspection** (#224)
   - `ZVec::getIoBackendType()`, `ZVec::getIoBackendTypeName(int $type)` and `ZVec::getIoBackendDescription()` report the I/O backend zvec picked for DiskANN disk reads. Linux tries `io_uring`, then `libaio`, and falls back to synchronous `pread()`; macOS always uses `pread()`. The value matters a lot for DiskANN throughput and previously could not be observed at all.
   - The choice is process-wide and resolved lazily, so the getters work without `ZVec::init()`. New constants `ZVec::IO_BACKEND_PREAD`, `IO_BACKEND_LIBAIO` and `IO_BACKEND_IO_URING` (`0`/`1`/`2`), matching the upstream C ABI.
@@ -98,13 +100,21 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Fixed
 
+- **The documented test command silently skipped most of the suite** (#215, #188)
+  - `php run-tests.php -n tests/` was documented as *the* way to run the suite, with the rationale that `-n` keeps a pre-installed legacy `zvec` extension from shadowing the FFI classes. That is only half the requirement. `-n` also strips `php.ini`, so wherever FFI is provided by a `conf.d` ini rather than compiled in, FFI disappears with it and **every** test reports `SKIP — reason: FFI extension not available`.
+  - Measured on a glibc host with PHP 8.5.4: 16 passed, **172 skipped**, 1 failed — and the run still printed a green-looking summary. A large skip count is a broken run, not a pass.
+  - The suite now needs both halves: `php run-tests.php -n -d extension=ffi.so tests/`. `-n` disables the legacy extension, `-d extension=ffi.so` keeps FFI available afterwards. If your PHP has FFI compiled in, drop the `-d` flag.
+  - `AGENTS.md` and `build_zvec.sh` updated, with a note that `Tests skipped` must be `0` before a run is trusted.
+  - `.github/scripts/run-tests.sh` already did the right thing and needed no change; the problem was the documentation only.
+
 - **`fetch_zvec_sdk.sh` picked the musl SDK on glibc hosts** (#215)
   - The musl check was `ls /lib/ld-musl-*.so.1`, which also succeeds on a glibc machine that merely has `musl-tools` installed. The musl SDK was then unpacked and every test failed at `ZVec::ffi()` with `libc.musl-x86_64.so.1: cannot open shared object file`.
   - The detection now inspects the loader of the binary that will actually load the SDK (`ldd $(command -v php)`), falling back to `/bin/sh`, and assumes glibc when neither marker is present. Override the probe with `PHP_BINARY`.
 
 - **Crash at exit and lost `ZVec::init()` settings with a shared libzvec** (#215)
   - libzvec exports the `GlobalConfig` singleton but keeps its init guard local, so calling the inline `GlobalConfig::Instance()` from our module built a second copy: `init()` settings were silently reset and the object was destroyed twice at exit (`free(): chunks in smallbin corrupted`, SIGABRT). The FFI adapter and the extension now resolve the instance from libzvec via `dlsym`. Test: `tests/bug_0056.phpt`.
   - Release builds of the Linux FFI adapter embed libstdc++; its symbols are now kept private (`-Wl,--exclude-libs,ALL`). Before, the dynamic linker bound part of them to the system libstdc++ that PHP already loads through libxml2/libicu, and `schema()` segfaulted inside `IndexParams::to_string()`.
+
 - **`run-tests.php` still overwrote hand-written `tests/<name>.php` files** (#187) — the temp copy of the extracted test used the old path; it now uses the `.php.tmp-extract` suffix too.
 - **`ZVecFieldSchema::getIndexType()` swapped VAMANA and DISKANN** — the C++ enum order differs from the PHP constants; the value is now mapped back explicitly. Test: `tests/bug_0055.phpt`.
 
@@ -129,30 +139,32 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   - Note: a second `set*Params()` call no longer implicitly resets previously set radius/linear/refiner (merge semantics — set `setRadius(0.0)` to restore the default).
   - Added regression test `test_query_params_order.phpt`.
 
-### Fixed
-
 - **Random rotation for INT8/INT4 quantization** (#177)
   - `ZVecIndexParams::setQuantizerEnableRotate(bool)` (fluent) enables random rotation before INT8/INT4 quantization for HNSW, Flat, IVF, and Vamana indexes — reduces quantization error and improves recall on quantized indexes.
   - Mirrors upstream zvec v0.6.0 `QuantizerParam(enable_rotate)` (C API: `zvec_index_params_set_quantizer_enable_rotate`).
 
-### Fixed
-
 - **`queryVector()` with Vamana / HNSW RaBitQ query params** (#193)
   - `ZVecVectorQuery::setVamanaParams()` and `setHnswRabitqParams()` previously created an `HnswQueryParams` object, so `queryVector()` on Vamana/RaBitQ indexes was always rejected by the engine (`query params type does not match the index type of vector field`).
   - FFI: added `zvec_vector_query_set_vamana_ef_search()` (`VamanaQueryParams`) and `zvec_vector_query_set_hnsw_rabitq_ef()` (`HnswRabitqQueryParams`), mirroring `apply_query_params`; the legacy `query()` path was unaffected.
   - Added regression tests `test_vector_query_vamana_params.phpt` and `test_vector_query_rabitq_params.phpt` (RaBitQ runs on Linux x86_64 only).
 
-### Fixed
-
 - **Deprecated index creation warnings** (#169)
   - The FFI bindings’ `createHnswIndex()`, `createHnswRabitqIndex()`, `createFlatIndex()`, and `createIvfIndex()` now emit `E_USER_DEPRECATED` before delegating to the unified `createIndex()` API.
+
 - **Regression gate against zvec v0.6.0** (#175)
   - Full `.phpt` suite verified (FFI mode, `-n`): 0 failures, only expected XFAILs (VECTOR_FP64, upstream-blocked) and the expected RaBitQ platform SKIP (Linux x86_64 only)
   - All 12 legacy `tests/bug_*.php` scripts pass against v0.6.0; cleaned-up statuses for `bug_0005_cleanup_after_failed_ops.php` and `bug_0006_rocksdb_lock.php` (no issues observed on v0.6.0)
   - `test_buffer_retry.phpt` now actually runs (previously it always skipped: the SKIPIF checked `method_exists()` before loading the library — fixed by requiring `src/ZVec.php` first)
   - Legacy `test_installer_platform.php` accepts the `macOS` label prefix on Darwin, matching its `.phpt` counterpart
   - Documented the `-n` requirement (pre-installed legacy extension shadows FFI classes — #188) and the `run-tests.php` legacy-file deletion hazard (#187) in `AGENTS.md` / `README.md`
 
+### Known issues
+
+- **Cross-module `operator new` / `operator delete` mismatch with the prebuilt SDK** (#228)
+  - valgrind reports `Mismatched free() / delete` where objects allocated by `operator new` inside `libzvec.so` are released by `operator delete(void*, unsigned long)` in `libzvec_ffi.so` (seen in `zvec_collection_insert` and `zvec_schema_free`). The SDK carries its `operator new` / `operator delete` as local symbols, so it uses a different allocation routine than the adapter, which shares the dynamic `libstdc++` with PHP.
+  - This is **not** the cause of the exit-time crash fixed above; that one was the duplicate `GlobalConfig` singleton. This one is currently benign — both paths end in the same glibc `free()` on a glibc-allocated chunk, and the full suite passes — but it is undefined behaviour and could corrupt the heap if the two implementations ever diverge (hardened allocators, jemalloc/tcmalloc preloaded under PHP).
+  - Not yet fixed. Candidate approaches are listed in #228; the preferred one is to make the adapter match the SDK's allocation routine. Verified on Linux x86_64 with the glibc SDK; not reproducible against a source build.
+
 ## [0.6.0] - 2026-07-30
 
 Adapter-only release: the zvec dependency moved from v0.5.x to v0.6.0. **The

diff --git a/MIGRATION.md b/MIGRATION.md
@@ -1,3 +1,12 @@
+# Migration Guide
+
+Two guides live in this file, each covering one upgrade:
+
+- **[v0.6.0 → v0.7.0](#migration-guide-v060--v070)** (at the bottom) — new
+  additive API, the prebuilt zvec SDK, and the test-suite invocation change.
+- **[v0.4.x → v0.5.0](#migration-guide-v04x--v050)** (below) — replacing the
+  deprecated v0.4.x APIs with the modern ones.
+
 # Migration Guide: v0.4.x → v0.5.0
 
 This guide helps users of the deprecated v0.4.x APIs migrate to the modern
@@ -435,3 +444,195 @@ creation fails at runtime with `DiskAnn factory entries are not registered`.
 `core_framework` and `core_knn_cluster` are deliberately *not* listed:
 `libzvec_core.a` already contains those objects and naming them duplicates
 every shared symbol.
+
+---
+
+# Migration Guide: v0.6.0 → v0.7.0
+
+Everything below is **additive**. No existing method was removed, renamed or
+retyped, so existing code keeps working unchanged. The one change that can
+affect you is how the test suite is invoked (see the last section), and the
+switch to a prebuilt zvec SDK.
+
+## New: a scalar index can be declared in the schema
+
+Previously an FTS or invert index on a STRING field needed two steps: create
+the collection, then call `createIndex()`. The index can now be part of the
+schema, so it exists from the first insert.
+
+```php
+// BEFORE — two steps, index absent until createIndex():
+$schema = new ZVecSchema('docs');
+$schema->addString('body');
+$collection = ZVec::create($path, $schema);
+$collection->createIndex('body', ZVecIndexParams::forFts());
+
+// AFTER — one step:
+$schema = new ZVecSchema('docs');
+$schema->addString('body', indexParams: ZVecIndexParams::forFts());
+$collection = ZVec::create($path, $schema);
+```
+
+The new argument is last and optional, so `addString('body')` and
+`addString('body', withInvertIndex: true)` are unchanged. Two notes:
+
+- Passing both `$withInvertIndex` and `$indexParams` now throws
+  (`Use either $withInvertIndex or $indexParams, not both`) instead of one
+  silently winning.
+- A mismatched index type is reported by `ZVec::create()` naming the field, e.g.
+  `scalar field[body] does not support vector index params`. That is unchanged
+  behaviour, just now reachable without a second call.
+
+## New: IVF-RaBitQ index type
+
+An IVF partitioned index storing RaBitQ-quantized vectors. Mirrors the Python
+`IvfRabitqIndexParam` / `IvfRabitqQueryParam`.
+
+```php
+$collection->createIndex('v', ZVecIndexParams::forIvfRabitq(
+    metricType: ZVecSchema::METRIC_L2,
+    nList: 1024,
+    totalBits: 7,      // 1..9
+    sampleCount: 0,
+));
+
+$query = (new ZVecVectorQuery('v', $vector))
+    ->setTopk(10)
+    ->setIvfRabitqParams(nprobe: 16);
+```
+
+New constants `ZVec::INDEX_TYPE_IVF_RABITQ` and `ZVec::QUERY_PARAM_IVF_RABITQ`,
+both `7`.
+
+**Platform:** upstream supports RaBitQ on **Linux x86_64 with AVX2+FMA or
+AVX-512 only**, and rejects an FP64 field, a dimension outside 64–4095, or a
+metric other than L2/IP/COSINE. On other platforms `createIndex()` fails with
+`NOT_SUPPORTED`. Note that `QUANTIZE_RABITQ` is *not* what selects this index —
+it is implied by the index type, and passing it to plain `forIvf()` is rejected.
+
+## New: Vamana `two_pass_build` and query prefetch
+
+`two_pass_build` runs a second full-graph Vamana construction pass: better graph
+quality, slower build. It is a trailing optional parameter, so existing
+positional calls to `forVamana()` are unaffected.
+
+```php
+$params = ZVecIndexParams::forVamana(
+    metricType: ZVecSchema::METRIC_IP,
+    maxDegree: 64,
+    // ...
+    twoPassBuild: true,
+);
+```
+
+`setVamanaPrefetch()` tunes search-time software prefetch, the counterpart of
+the existing `setHnswPrefetch()`. A `prefetchOffset` of `0` disables prefetch;
+`0` lines means "derive from vector size". The call order relative to
+`setVamanaParams()` does not matter.
+
+```php
+$query = (new ZVecVectorQuery('v', $vector))
+    ->setTopk(10)
+    ->setVamanaPrefetch(256, 4);
+```
+
+> **Known limitation:** prefetch — HNSW *and* Vamana — is honoured only by
+> `queryVector()`. The legacy `query()` path sends just `queryParamType`,
+> `ef`/`nprobe`, `radius`, `isLinear` and `isUsingRefiner` upstream, so prefetch
+> is silently dropped there. This is not new in v0.7.0; it applies equally to
+> `setHnswPrefetch()`.
+
+## New: `ZVec::init()` tuning options
+
+```php
+ZVec::init(
+    // ...existing parameters...
+    ?string $jiebaDictDir = null,             // folder with jieba.dict.utf8 + hmm_model.utf8
+    ?float $ftsBruteForceByKeysRatio = null,  // 0.0-1.0, upstream default 0.05
+);
+```
+
+Read back with `ZVec::getJiebaDictDir()` and
+`ZVec::getFtsBruteForceByKeysRatio()`. With neither option given, the jieba
+dictionary shipped in `zvec_data/jieba_dict` is still found automatically.
+
+Like every other `init()` option, both take effect only on the **first**
+successful `init()` in a process; upstream ignores later calls. `null` is the
+"keep the upstream default" marker, because `0.0` is a real value upstream
+accepts.
+
+Both are validated in PHP before any FFI call, which is the only place the
+check can live: upstream `GlobalConfig::initialize()` sets its initialized flag
+*before* validating, so a rejected value would leave the library marked
+initialized and make every later `init()` a silent no-op. A `jiebaDictDir`
+missing its dictionary files is rejected too, for a worse reason — cppjieba
+calls `abort()`, killing the process with exit code 134 rather than raising a
+catchable error.
+
+## New: DiskANN I/O backend introspection
+
+zvec picks an I/O backend for DiskANN disk reads on first use — on Linux it
+tries `io_uring`, then `libaio`, then falls back to synchronous `pread()`;
+macOS always uses `pread()`. The choice dominates DiskANN throughput and was
+previously unobservable.
+
+```php
+ZVec::getIoBackendType();          // ZVec::IO_BACKEND_PREAD | LIBAIO | IO_URING
+ZVec::getIoBackendTypeName(2);     // "io_uring"
+ZVec::getIoBackendDescription();   // human-readable, with install hints on Linux
+```
+
+New constants `ZVec::IO_BACKEND_PREAD` / `IO_BACKEND_LIBAIO` /
+`IO_BACKEND_IO_URING` (`0` / `1` / `2`). The getters work without
+`ZVec::init()` — the backend is resolved lazily and the value is process-wide.
+
+If you see `pread` on Linux, neither io_uring nor libaio could be loaded, so
+DiskANN reads are synchronous. Installing libaio (`libaio1t64` on Ubuntu
+24.04+) enables async I/O.
+
+## Changed: the test suite must be run with FFI enabled
+
+This is the one change that can break someone's workflow.
+
+```bash
+# BEFORE — silently skipped most of the suite on many machines:
+php run-tests.php -n tests/
+
+# AFTER:
+php run-tests.php -n -d extension=ffi.so tests/
+```
+
+`-n` keeps a pre-installed legacy `zvec` extension from shadowing the FFI
+classes, but it also strips `php.ini` — so wherever FFI is provided by a
+`conf.d` ini rather than compiled in, FFI disappears with it. Measured on a
+glibc host with PHP 8.5.4: 16 passed, **172 skipped**, 1 failed, and the run
+still printed a green-looking summary.
+
+If your PHP has FFI compiled in, drop the `-d extension=ffi.so` flag. Either
+way, check that `Tests skipped` is `0` before trusting a run — a large skip
+count means FFI was not loaded, not that the code is fine.
+
+## Changed: zvec is now a prebuilt SDK
+
+zvec is no longer compiled from source. `./fetch_zvec_sdk.sh` downloads the
+official prebuilt SDK from the
+[alibaba/zvec releases](https://github.com/alibaba/zvec/releases) (pinned
+SHA-256) and `./build_zvec.sh` builds only our adapter.
+
+If you build from source or vendor `libzvec`, the important consequence is
+that `libzvec` and `zvec_data/` (the jieba dictionary) must sit in the same
+directory as `libzvec_ffi`. The adapter resolves libzvec via an `$ORIGIN` /
+`@loader_path` rpath, so a Composer install under `lib/` needs all three.
+
+Release asset names changed: `libzvec_ffi-<platform>.tar.gz` instead of
+`libzvec_ffi-ubuntu24-x86_64.tar.gz`. `vendor/bin/zvec-install` handles this
+automatically; only custom CI that fetches assets by name needs updating.
+
+## Unchanged
+
+- Every pre-v0.7.0 method, constant and behaviour.
+- All previously deprecated `create*Index()` and `addField*()` methods still
+  work and still emit `E_USER_DEPRECATED`.
+- `queryById()`, `queryMulti()`, `queryWithReranker()` and
+  `groupByQuery()` signatures.
+