feat(collection): full-collection scans via iterDocs() / ZVecDocIterator (#226)

[s2x]

Closes #226. Depends on #222, now merged as #238.

Summary

There was no way to read every document in a collection. fetch() needs primary keys, queryByFilter() needs both a filter and a topk, and neither can walk a whole collection — so export, backup and migration were not possible at all.

foreach ($collection->iterDocs(outputFields: ['id', 'name'], includeVector: false) as $pk => $doc) {
    // ...
}

Backed by upstream's Collection::create_iterator() (zvec v0.7.0), exposed the way Python does with iter_docs(). Constant memory regardless of collection size, over an isolated snapshot taken at call time.

Ownership is the whole difficulty here

The native iterator holds a std::shared_ptr to the collection, copied out of the collections registry at creation. Two consequences:

Upstream's release_slot captures a raw Collection*. Freeing the handle underneath a live iterator would make it release a slot in freed memory. The shared_ptr makes that impossible: zvec_collection_free() only drops our reference, and the C++ collection stays alive until the last iterator goes.

Member order in DocIteratorHolder is load-bearing. Members are destroyed in reverse declaration order, so collection is declared first and iterator last — the iterator releases first (returning its slot), the collection second. The other order deadlocks: upstream's ~CollectionImpl waits on a condition variable for every iterator to close, which in a single-threaded PHP process means waiting for something that can never happen.

That failure mode is a hang, not a crash, so no ordinary test would catch it. Hence tests/test_doc_iterator_shutdown.phpt as its own file: the script ends with both a collection and an open iterator alive, and any wrong release order shows up as an infinite loop rather than a failing assertion.

On the PHP side ZVecDocIterator holds a reference to its ZVec object, so the collection cannot be collected out from under a live iterator.

The upstream guards are surfaced, not worked around

While an iterator is open, close(), destroy(), schema DDL and optimize() return FAILED_PRECONDITION and change nothing. That only works because of #222 — before it, a rejected destroy() freed the native collection and the next call was a use-after-free. Writes, flush() and queries are unaffected, and the test asserts that too.

Two smaller decisions

\Iterator, not IteratorAggregate returning a Generator. A second getIterator() call on the same object would silently yield nothing from the exhausted handle. rewind() failing loudly is better, and close() / isClosed() live on the same object.

Nullable normalization is limited to the requested fields. fetch() has returned unset nullable fields as explicit null since #192, and iteration matches — but only for fields actually asked for. Otherwise hasField('weight') would be true under outputFields: ['id'], which contradicts upstream's own behaviour.

Verification

The lifecycle test covers the whole matrix, including the two cases that would be silent failures otherwise:

destroy: FAILED_PRECONDITION
usable after destroy: 1            # the handle stayed valid — this is #222 working
close: FAILED_PRECONDITION
usable after close: yes
ddl: FAILED_PRECONDITION
optimize: FAILED_PRECONDITION
write while open: 1                # writes are not blocked
drained: 10
close after drain: ok              # exhaustion returned the slot
iter after unset: 11               # collection variable dropped mid-iteration
destruct with open iterator: ok
closed: Collection is closed. Open with ZVec::open() to continue.
iter after destruct: 11            # iterator still works
reopen: ok                         # lock was released
destroy after release: ok

Full suite:

Number of tests : 208               208
Tests skipped   :   0 (  0.0%)
Tests failed    :   0 (  0.0%)
Tests passed    : 206 ( 99.0%)

Four new tests plus extensions to test_ffi_load.phpt and test_null_handle_collection.phpt. test_dbs/ is left with only .gitignore.

Note

Two values in the lifecycle test are 11 rather than the 10 of the earlier iterator, because a document written after the first snapshot is visible to later ones. That is commented inline so the asymmetry does not look like an off-by-one.

Out of scope, per the issue: filtered iteration (upstream has no filter in IteratorOptions), resumable iteration, multi-threaded iteration, and the pre-existing char*[N] leak in toCStringArray() for other callers — iterDocs() frees its own array.

🤖 Generated with Claude Code