Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
Expand Up @@ -23,3 +23,7 @@
*.db binary
*.png binary
*.ison text

# Byte-exact fixtures: the snapshot stores the sidecar's length and a hash of its bytes, so
# any line-ending rewrite on checkout would make it (correctly) stale.
tests/fixtures/snapshot/* -text
7 changes: 6 additions & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -331,6 +331,11 @@ jobs:
# in each file and requires every port to raise its mismatch error. The
# script builds the Rust, C#, C++ and TypeScript helpers outside the
# checkout (scripts/cross_port/{rust,csharp,cpp}) and prints a PASS/FAIL
# table; any failing pair fails the job.
# table; any failing pair fails the job. Each writer also writes the
# optional binary snapshot (<graph>.embeddings.snap), and three matrices
# run: every reader must accept every writer's snapshot (source must be
# 'snapshot'); every reader must still raise its mismatch error for an
# edited graph with a stale snapshot beside it; and with the snapshot
# deleted the JSON path must give the same observation.
- name: Graph files and sidecars, all six ports, all pairs
run: python scripts/cross_port/graph_files.py
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -35,3 +35,4 @@ obj/
.idea/
.DS_Store
Thumbs.db
!tests/fixtures/snapshot/*.embeddings.json
54 changes: 50 additions & 4 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -208,6 +208,51 @@ JavaScript, Rust and C++.
Costs: single-thread exact scan is 0-13% slower, ingest is slower, and
updates at 1M are 2-3x slower because index upkeep moved from the first
search to the write.
- **Python, C#, TypeScript, JavaScript and C++: one copy of every vector,
batch search, binary snapshot** (the Rust design of `DESIGN.md` sections 1, 5
and 11 carried over; measured per port below). Each store now keeps every
vector once in a chunked (or contiguous) float32 table and records refer to
its rows, so the second copy that the index or the scan cache used to hold is
gone, and pending rows fold into the IVF index at write time (JavaScript /
TypeScript keep their search-time fold: a fold inside a write would block it
for seconds). Memory at 100k x 384 (raw 146 MB, steady RSS above baseline,
3 runs, shared laptop): Python 437 -> 211 MB, TypeScript / JavaScript 435 ->
197 MB (590 -> 203 MB with the index), C# IVF store 409 -> 215 MB, C++ IVF
store 329 -> 167 MB; at 1M x 384 (one run each) the new figures are 1.12-1.25x
raw (Python 1.25x, TypeScript 1.21x, C# 1.15x exact / 1.18x with IVF, C++
1.12x with IVF; JavaScript was not measured at 1M);
the old 3.3-7.0 GB figures are in `benchmarks/competitive/`. New:
`similarity_search_batch` (Python: one matrix-matrix product; 8x one-at-a-time
at 100k), `SimilaritySearchBatch` (C#), `similaritySearchBatch` (C++),
parallel across queries and bit-identical to single-query search in C# and
C++; searches are safe to run concurrently in C# and C++. The optional
`.embeddings.snap` file (`save(snapshot=True)` / `Save(..., snapshot: true)` /
`saveGraph(..., {snapshot: true})`, byte layout in Rust `DESIGN.md` section
11) is written and read by all six ports, is 3-5x smaller and roughly
13-45x faster to save and load (C#, C++, Python) than the JSON sidecar at 100k x 384, and is the only
way to persist a 100k-row store in JavaScript (the JSON export exceeds V8's
string limit). A snapshot is used only when it provably belongs to the graph
and sidecar on disk, otherwise the JSON is read; graph files, sidecar bytes
and digests are unchanged. Cross-port: a snapshot written by any port loads
in every other port (`scripts/cross_port/graph_files.py`).
**Breaking, per port:** `get` / `get_embedding` return a copy of the vector
in C#, C++ (`std::optional<EmbeddingRecord>`), TypeScript and JavaScript;
`AnnStats.vectorBytes` / `cacheBytes` are 0 (the index owns no vectors);
pending rows fold below 256 at write time (Python, C#, C++);
TypeScript / JavaScript store vectors as float32 (inputs that are not
float32-exact read back rounded: `0.1` -> `0.10000000149011612`; scores
changed by at most 1.3e-8 in a 1,500-vector check); C# `Add` copies the
caller's array; Python in-memory stores no longer have a SQLite table; equal
scores order by arrival in C# and C++. **Python exact top-k scores are now
canonical:** BLAS narrows the field and the rows near the k-th score are
rescored with one kernel, so batch equals single bit for bit; scores can
differ from the previous release in the last float32 bit and near-ties can
reorder (`store.canonical_scores = False` restores the old behaviour).
Costs: single-query IVF is within noise in Python and C++, about 9% slower in
TypeScript / JavaScript and 8-25% slower in C# (its scalar kernel keeps C#'s
own summation order, so C# scores can still differ from Rust's in the last
bits, as before); C++ IVF writes cost about 5 us more because of the
fold.
- **Async encoders in TypeScript / JavaScript** - `AsyncEncoder` and
`...Async` methods (`addNodeAsync`, `updateNodeAsync`, `embedAllNodesAsync`,
`reembedStaleAsync`, `similaritySearchAsync`, `semanticMultiHopAsync`, ...)
Expand Down Expand Up @@ -242,15 +287,16 @@ JavaScript, Rust and C++.
minutes at 1M vectors).
- **Not a vector-database replacement.** Against faiss, hnswlib and usearch
the ports are 2-4x slower per query at 100k and about 14x slower at 1M text
vectors at equal recall; only Rust has batched and concurrent search (the other ports serialise on one lock); vectors
persist as JSON only; and memory is 2-4x higher (Rust is now 1.1x raw; the other ports still store vectors twice). Graph-aware retrieval
vectors at equal recall; only Rust, C# and C++ have concurrent search and Python has batch search (TypeScript and JavaScript are single-threaded); the ANN index is
never persisted and the binary snapshot is opt-in (JSON is the portable default); and memory is 1.12-1.25x raw at 1M in the four ports measured, against 1.6-1.9 GB for faiss / hnswlib at 1M x 384 (re-run `benchmarks/competitive/` before quoting the
ratios: they predate the rework). Graph-aware retrieval
beating plain vector search on multi-hop questions is the supported claim,
and it depends on the graph carrying real structure
(`benchmarks/multihop/README.md`).
- Fast filtered search was only measured for node-type filters, and only Rust
is fast there (Python's filtered search is no faster than its unfiltered
scan). The C++ exact scan is scalar and about 6x slower than numpy; the C#
and JS exact scans are also slower than numpy.
scan). The C++, C# and JS exact scans are scalar and still slower than numpy
(about 2x faster than before this change in C++ and C#, 1.6x in JS).
- The digest cannot tell `1` from `1.0` in JavaScript / TypeScript, and
integers beyond 64 bits are not supported in C# / C++ / Rust. Where a graph
contains such values the loader warns instead of raising a false mismatch.
Expand Down
15 changes: 15 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -322,6 +322,21 @@ graph = SemanticGraph.load("social.isong", encoder=my_encoder)
Pass the `encoder=` you saved with. Without it, the first text search on a
loaded graph constructs a `SentenceTransformerEncoder` and downloads a model.

**Large stores: the optional binary snapshot.** The JSON sidecar is portable but
big and slow at scale (450-810 MB and 4-16 s to write at 100k x 384 vectors, by port;
past that, JavaScript cannot build the string at all). `graph.save(path,
snapshot=True)` also writes `<path>.embeddings.snap`, a checksummed binary copy
of the same embeddings that is about 3-5x smaller and an order of magnitude
faster to save and load. `load` uses it only when it provably belongs to the
graph and sidecar on disk, and otherwise reads the JSON, so the JSON stays the
portable default and nothing about the graph file or its digest changes.
`load_with_report` says which one it used and why a snapshot was skipped. Every
port reads and writes the same file (format: `isongraph-vector-rs/DESIGN.md`
section 11); the ANN index is not in it and is rebuilt after load. Snapshot
spelling per port: `save(path, snapshot=True)` (Python), `saveGraph(graph, path,
{ snapshot: true })` (TypeScript / JavaScript), `Save(path, snapshot: true)` (C#),
`save(path, "auto", true, true)` (C++), `save_with_snapshot(path, None)` (Rust).

The other five ports persist embeddings to SQLite and exchange the portable
payload, but the graph itself stays in memory there.

Expand Down
15 changes: 9 additions & 6 deletions ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,18 +14,21 @@ What is deliberately not done yet, in rough priority order. Items marked
(`isongraph-vector-rs/DESIGN.md`, `benchmarks/rust-stage-c/`): concurrent
and batch search, one copy of the vectors, no post-write stall, opt-in HNSW
(level with faiss HNSW / hnswlib single-thread), int8 scan, binary snapshot.
Still open: port the proven pieces to the other ports; persist the ANN
index; faster HNSW single-call inserts (1.4k/s at 1M); exact scan is at
One copy of the vectors, batch search and the binary snapshot are now in
every port (HNSW and int8 are still Rust-only). Still open: HNSW for C++ and
C#; persist the ANN index; faster HNSW single-call inserts (1.4k/s at 1M); exact scan is at
parity with faiss Flat, not ahead.
5. **Fix the slow ports.** The C++ exact scan has no scan cache and is scalar
(6x slower than numpy); JavaScript k-means takes about 8 minutes at 1M.
5. **Fix the slow ports.** The C++, C# and JS exact scans are scalar and still
slower than numpy; JavaScript k-means takes about 8 minutes at 1M.
6. **Per-record encoder version.** Today a version-only change warns on
load but cannot be fixed by `reembed_stale`.

## Later

- **Binary persistence** for vectors and the index, as an addition to the
portable JSON sidecar (which stays the default and keeps the digest).
- **Persist the ANN index** next to the binary snapshot (the snapshot itself
exists in every port; the index is rebuilt after load).
- **Snapshot format v2:** an id-kind bit per row, so ports with integer ids need
no graph lookup to restore them.
- **Approximate search for the standalone SQLite stores.**
- **Relationship-type weights** (per type rather than a per-edge property),
and richer node predicates than a callback.
Expand Down
221 changes: 221 additions & 0 deletions benchmarks/stage-d-python/bench_stage_d.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,221 @@
#!/usr/bin/env python3
"""
Stage D benchmark for the Python port: memory, single-query and batch search,
snapshot against JSON.

Each measurement runs in a fresh child process (RSS is only meaningful there)
and is repeated ``--runs`` times; the parent prints medians and ranges.

python benchmarks/stage-d-python/bench_stage_d.py --impl new --runs 3
python benchmarks/stage-d-python/bench_stage_d.py --impl path/to/old/__init__.py --runs 3

``--impl new`` (default) is this checkout's package; any other value is the path
of an ``__init__.py`` to load instead (used to compare against the code as it was
before Stage D). Data are random unit vectors (fixed seed), 5,000 per add_batch
call as the competitive benchmark does. Timings are indicative only when the
machine is shared; RSS figures are not affected by that.
"""

import argparse
import gc
import importlib.util
import json
import os
import statistics
import subprocess
import sys
import tempfile
import time

HERE = os.path.dirname(os.path.abspath(__file__))
REPO = os.path.abspath(os.path.join(HERE, "..", ".."))


def load_impl(spec: str):
path = os.path.join(REPO, "isongraph_vector-py", "__init__.py") if spec == "new" else spec
name = "isongraph_vector_bench_" + str(abs(hash(path)))
module_spec = importlib.util.spec_from_file_location(
name, path, submodule_search_locations=[os.path.dirname(path)])
module = importlib.util.module_from_spec(module_spec)
sys.modules[name] = module
module_spec.loader.exec_module(module)
return module


def rss_mb(proc) -> float:
return proc.memory_info().rss / 2**20


def child(args) -> dict:
import numpy as np
import psutil

mod = load_impl(args.impl)
proc = psutil.Process()
gc.collect()
base = rss_mb(proc)
out = {"impl": args.impl, "n": args.n, "dim": args.dim, "baseline_mb": round(base, 1),
"raw_mb": round(args.n * args.dim * 4 / 2**20, 1)}

kwargs = {}
if args.ann:
kwargs["ann"] = "ivf"
db_dir = tempfile.TemporaryDirectory() if args.file_db else None
db_path = os.path.join(db_dir.name, "emb.db") if db_dir else ":memory:"
store = mod.EmbeddingStore(db_path, dimension=args.dim, **kwargs)
rng = np.random.default_rng(1234)
centers = None
if args.clustered: # 256 cluster centres + noise: the shape of real embeddings, for IVF recall
centers = rng.standard_normal((256, args.dim)).astype(np.float32)
centers /= np.linalg.norm(centers, axis=1, keepdims=True)

def draw(count, gen):
if centers is None:
return gen.standard_normal((count, args.dim)).astype(np.float32)
which = gen.integers(0, 256, count)
return centers[which] + 0.5 * gen.standard_normal((count, args.dim)).astype(np.float32) / np.sqrt(args.dim)

started = time.perf_counter()
for start in range(0, args.n, 5000):
count = min(5000, args.n - start)
block = draw(count, rng)
block /= np.linalg.norm(block, axis=1, keepdims=True)
store.add_batch([(("doc", start + i), "", v) for i, v in enumerate(block.tolist())])
del block
out["ingest_s"] = round(time.perf_counter() - started, 2)
gc.collect()
out["rss_after_ingest_mb"] = round(rss_mb(proc) - base, 1)

qrng = np.random.default_rng(99)
queries = draw(512, qrng)
queries /= np.linalg.norm(queries, axis=1, keepdims=True)
qlist = queries.tolist()

started = time.perf_counter()
store.similarity_search(qlist[0], 10)
out["first_query_s"] = round(time.perf_counter() - started, 3)
gc.collect()
out["rss_steady_mb"] = round(rss_mb(proc) - base, 1)
out["rss_ratio_to_raw"] = round(out["rss_steady_mb"] / out["raw_mb"], 2)

if args.ann:
started = time.perf_counter()
store.build_ann_index()
out["ivf_build_s"] = round(time.perf_counter() - started, 2)
gc.collect()
out["rss_with_ivf_mb"] = round(rss_mb(proc) - base, 1)

for q in qlist[:20]:
store.similarity_search(q, 10)
timed = qlist[:args.queries]
started = time.perf_counter()
for q in timed:
store.similarity_search(q, 10)
elapsed = time.perf_counter() - started
out["single_qps"] = round(len(timed) / elapsed, 1)
out["single_ms"] = round(1000 * elapsed / len(timed), 3)
if args.ann:
report = store.recall_check(qlist[:50], k=10)
out["ivf_recall_at_10"] = round(report.recall, 4)

if hasattr(store, "similarity_search_batch") and not args.ann:
for batch in (16, 64, 256):
best = None
for _ in range(3):
started = time.perf_counter()
store.similarity_search_batch(qlist[:batch], 10)
took = time.perf_counter() - started
best = took if best is None else min(best, took)
out[f"batch{batch}_qps"] = round(batch / best, 1)
# the same queries one at a time, for a like-for-like ratio
started = time.perf_counter()
for q in qlist[:256]:
store.similarity_search(q, 10)
out["single_256_qps"] = round(256 / (time.perf_counter() - started), 1)

if args.files and not args.ann:
with tempfile.TemporaryDirectory() as tmp:
json_path = os.path.join(tmp, "e.embeddings.json")
started = time.perf_counter()
store.save_embeddings(json_path)
out["json_save_s"] = round(time.perf_counter() - started, 2)
out["json_mb"] = round(os.path.getsize(json_path) / 2**20, 1)
gc.collect()
other = mod.EmbeddingStore(":memory:", dimension=args.dim)
started = time.perf_counter()
other.load_embeddings(json_path)
out["json_load_s"] = round(time.perf_counter() - started, 2)
assert other.count() == args.n
other.close()
del other
gc.collect()
if hasattr(store, "save_snapshot"):
snap_path = os.path.join(tmp, "e.embeddings.snap")
started = time.perf_counter()
store.save_snapshot(snap_path)
out["snap_save_s"] = round(time.perf_counter() - started, 2)
out["snap_mb"] = round(os.path.getsize(snap_path) / 2**20, 1)
other = mod.EmbeddingStore(":memory:")
started = time.perf_counter()
other.load_snapshot(snap_path)
out["snap_load_s"] = round(time.perf_counter() - started, 2)
assert other.count() == args.n
other.close()
store.close()
if db_dir:
db_dir.cleanup()
return out


def parent(args) -> None:
runs = []
for i in range(args.runs):
cmd = [sys.executable, os.path.abspath(__file__), "--child", "--impl", args.impl,
"--n", str(args.n), "--dim", str(args.dim), "--queries", str(args.queries)]
if args.ann:
cmd.append("--ann")
if args.files:
cmd.append("--files")
if args.clustered:
cmd.append("--clustered")
if args.file_db:
cmd.append("--file-db")
done = subprocess.run(cmd, capture_output=True, text=True)
if done.returncode != 0:
sys.stderr.write(done.stderr)
raise SystemExit(done.returncode)
runs.append(json.loads(done.stdout.strip().splitlines()[-1]))
print(f"run {i + 1}/{args.runs}: {runs[-1]}", flush=True)
keys = [k for k, v in runs[0].items() if isinstance(v, (int, float)) and k not in ("n", "dim")]
summary = {}
for k in keys:
values = [r[k] for r in runs if k in r]
summary[k] = {"median": statistics.median(values), "min": min(values), "max": max(values)}
print(f"== impl={args.impl} ann={args.ann} clustered={args.clustered} file_db={args.file_db} n={args.n} dim={args.dim} runs={args.runs}")
for k, v in summary.items():
print(f"{k:24s} median {v['median']:>10} range [{v['min']} .. {v['max']}]")


def main() -> None:
ap = argparse.ArgumentParser()
ap.add_argument("--impl", default="new")
ap.add_argument("--n", type=int, default=100_000)
ap.add_argument("--dim", type=int, default=384)
ap.add_argument("--runs", type=int, default=3)
ap.add_argument("--queries", type=int, default=200)
ap.add_argument("--ann", action="store_true", help="measure the IVF index instead of the exact scan")
ap.add_argument("--files", action="store_true", help="also time JSON and snapshot save/load")
ap.add_argument("--file-db", action="store_true",
help="back the store with a SQLite file instead of :memory:")
ap.add_argument("--clustered", action="store_true",
help="clustered data instead of uniform random (uniform is the worst case for IVF recall)")
ap.add_argument("--child", action="store_true")
args = ap.parse_args()
if args.child:
print(json.dumps(child(args)))
else:
parent(args)


if __name__ == "__main__":
main()
Loading
Loading