Repository navigation
Stage D part 1: one vector copy, batch search and binary snapshot in the other five ports - #3
Open
maheshvaikri-code wants to merge 7 commits into
Open
maheshvaikri-code wants to merge 7 commits into
maheshvaikri-code wants to merge 7 commits into
Conversation
…r other ports' output snapshot_fixture writes tests/fixtures/snapshot/golden.* (graph, JSON sidecar, binary snapshot) with the awkward cases in it; every port must load it from the snapshot. snapshot_check loads a graph saved by any port and reports whether the snapshot was accepted and whether the store equals a JSON-only load. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…snapshot VectorTable holds each vector once in 1024-row chunks; the IVF index keeps only cell lists and follows table compaction; pending rows fold at write time. Searches share a ReaderWriterLockSlim and run concurrently; SimilaritySearchBatch runs queries in parallel with results bit-identical to the single-query path. Optional .embeddings.snap per Rust DESIGN.md section 11 (reproduces the Rust golden snapshot byte for byte). SaveEmbeddings streams instead of building one giant string. Fixes the competitive bench's recall_before 0.000 (a bench bug: reference recall unset when the sweep never reached 0.95). Breaking: Get/GetEmbedding return a copy of the vector, Add copies the caller's array, AnnStats.VectorBytes is 0, pending rows fold below 256 at write time. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ary snapshot VectorTable replaces the record map plus ScanCache: float32 rows in 1024-row chunks, norms/type codes/models in parallel columns, tombstones compacted at 25%, IVF reads rows from the table. The exact scan scores four rows at a time with independent per-row sums, so scores are unchanged and the scan is faster. Optional .embeddings.snap per Rust DESIGN.md section 11 (reproduces the Rust golden snapshot byte for byte; the only way to persist 100k x 384 in JS, since the JSON export exceeds V8's string limit). New browser-safe snapshot module with injected SHA-256; file I/O in node.ts / node.js. Breaking: vectors are stored as float32 (values widen to double on read), get()/getEmbedding() return a copy, AnnStats.cacheBytes is always 0. Not done: write-time IVF fold (kept the search-time fold; a fold in a write would block it for seconds), batch search (single-threaded). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… snapshot
In-memory stores no longer keep a SQLite copy next to the scan matrix; the
contiguous float32 matrix is the table (pure-Python path: one array('f') per
row, patched in place with tombstones and compaction). IVF pending rows fold
at write time (256). similarity_search_batch does one matrix-matrix product
per group of queries. Optional .embeddings.snap per Rust DESIGN.md section 11
(save(snapshot=True), load_with_report; verified accepted by the Rust reader).
Bounded exact search now returns canonical scores: BLAS only narrows the field
and every row within a derived error margin of the k-th score is rescored with
one kernel that gives the same bits whatever else is in the call, which is what
makes batch equal single bit for bit. Scores can differ from the previous
release in the last float32 bit and near-ties can reorder;
store.canonical_scores = False restores the old behaviour.
Breaking: in-memory stores have no SQLite table (_get_conn is empty), IVF
pending is below 256 between writes.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…, optional binary snapshot The index no longer copies vectors; ChunkedMatrix (1024-row chunks) is the only copy and records reference rows. One out-of-line scoring routine (detail::cosineRows, noinline/noclone) is used by the scan, batch, IVF rerank and snapshot-loaded stores. Searches share a shared_mutex; similaritySearchBatch scores tiles of 16 queries per chunk across bounded worker threads. Pending rows fold at write time (256). Optional .embeddings.snap per Rust DESIGN.md section 11 with a streaming SHA-256; re-saving the Rust golden store reproduces its snapshot byte for byte. Breaking: get()/getEmbedding() return std::optional<EmbeddingRecord> (a copy), EmbeddingRecord lost ann_slot, AnnStats::vectorBytes is 0, equal scores order by arrival, pending rows fold at the 256th write. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…larifications DESIGN.md section 11: row order is the writing port's sidecar order (readers must not rely on it), repeated-key and non-UTF-8 rejection spelled out, count limit for ports with bounded arrays, short-file fingerprint windows, and the fact that ids are text (ports with integer ids restore the kind from the graph; a v2 should carry an id-kind bit). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Every port writes the snapshot, every port must accept every other port's snapshot (source == snapshot), a stale snapshot beside an edited graph must not mask the mismatch error, and a copy without the .snap must load through the JSON with the same observation. 108/108 locally. Also repairs the C++ helper for getEmbedding returning std::optional. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stage D, part 1: port the proven Rust pieces to the other five ports
Carries three things from the reworked Rust crate (
isongraph-vector-rs/DESIGN.md) to Python, C#, TypeScript, JavaScript and C++. HNSW for C++/C# is part 2.1. One copy of every vector. Each store keeps each vector once in a chunked float32 table; records and the IVF index refer to its rows. Pending rows fold into IVF at write time (Python, C#, C++; TS/JS keep the search-time fold). Steady RSS at 100k x 384 (raw 146 MB): Python 437 -> 211 MB, TS/JS 435 -> 197 MB, C# IVF store 409 -> 215 MB, C++ IVF store 329 -> 167 MB. At 1M x 384 (one run each): 1.12-1.25x raw.
2. Batch / concurrent search. Python: matrix-matrix batch (about 8x one-at-a-time). C#, C++: concurrent searches plus parallel batch, bit-identical to single-query. TS/JS: none (single-threaded).
3. Optional binary snapshot (
<graph>.embeddings.snap, spec in RustDESIGN.mdsection 11). All six ports write and read it; the JSON sidecar stays the portable default and the graph file, sidecar bytes and digest are unchanged. A snapshot is used only when it provably belongs to the graph and sidecar on disk. 3-5x smaller and roughly 13-45x faster than JSON at 100k x 384; the only way to persist 100k rows in JS.Verification
tests/fixtures/snapshot/) is loaded from the snapshot by every port; C#, C++ and TS re-save it byte for byte.snapshot), a stale snapshot must not mask a mismatch error, and loads without the.snapmust match. 108/108 locally.Breaking changes
get/getEmbeddingreturn a copy in C#, C++ (std::optional<EmbeddingRecord>), TS and JS; index stats report 0 vector bytes.store.canonical_scores = Falserestores the old behaviour).Addcopies the caller's array.Not done / limits
recall_before 0.000(a benchmark bug).🤖 Generated with Claude Code